Build an outgoing webhook system for integrations #735

Open
timabbott opened this Issue Apr 29, 2016 · 13 comments

Projects

None yet

3 participants

@timabbott
Member

Incoming webhooks have been a great model for making it easy to add new integrations sending messages into Zulip. For applications to receive messages from Zulip, it'd be great to have an outgoing webhook system where a user can trigger events being sent to third party services from Zulip through some reasonable syntax.

https://api.slack.com/outgoing-webhooks is a reference on how Slack does this; it's not clear to me at first whether we want something that resembles that, or https://api.slack.com/slash-commands, or both, but we can likely share most of the code between both services.

@timabbott timabbott added this to the 2016 roadmap milestone Apr 29, 2016
@kernicPanel

Some thoughts about this.
I think there's no real functional differences between outgoing webhooks and slash commands.
In slack they only differ on their UI/UX implementation. Outgoing webhooks are regular messages send to another server. Slash commands populates an autocomplete list which I find more usable.
In the end, the data send are barely identical or could be.

I think the whole point here is what kind of UI and functionality do you want to offer to your users.
In my point of view slash commands autocomplete list is better. But it should be more flexible.
I would like to have a slash command system that allows autocompletion from an external service.
Let me know if you want me to explain this further, I already documented this idea for other OSS slack alternatives, and they are interested.
Unfortunately, I'm really not confortable with their tech stack, so I wasn't able to implement it.
I'm currently (re)building a slack bot supporting commands autocompletion in django (first version currently used in our team is written in nodejs)

Also with Zulip's stream system, and his original UI (regarding slack's conventions) I think it deserves a new way of offering a chatops sytem. I don't know if using regular streams for sending messages to a service is really useful. Maybe a sidebar or something else could be more appropriate.
But having a way to send a command response to a stream could be useful.

Sorry this post is not very well structured, please let me know if something is not clear.

@kernicPanel

Sorry, re-reading my previous post and slack doc, point me to errors.

There is a main diff between outgoing webhooks and slash commands.

OW can use a regular user's message to trigger something.
SC only triggers something and allows delayed responses.
SC also add a possibility to offer admin commands to users, like /join /leave /invite etc.

Both are useful, but I keep believing that extended SC as I described above will be great.
Also, I doubt that storing SC and his response in streams is really useful.

@timabbott
Member
timabbott commented Jun 28, 2016 edited

Since @rahuldeve expressed interest in working on this, here's a design proposal for outgoing webhooks:

  • We add outgoing webhooks as a new type of bot user (bot_type=OUTGOING_WEBHOOK). When a user configures one of these bots, they provide a URL to post to. We should probably plan to be able to extend this interface to enable setting alert_words for these bots (from a UI design perspective), though as I note below I wouldn't start with implementing the UI.
  • By default, they are triggered by @-mentioning the bot in a public stream, or sending a (group?) private message to the bot. To make this work, we will need to treat these bots as subscribed to all public streams for the purposes of the mentions system.
  • We modify some combination of Bugdown and do_send_messages to produce a list of webhook bots that are interested in a given message. This may require a bit of work to adjust the validation for whether users are subscribed (I'd prefer to not require outgoing webhook bots to subscribe to anything; ideally, they should just not be subscribable to streams).
  • For each such (message, bot) pair, do_send_messages delivers the relevant messages to a new RabbitMQ queue (zerver/lib/queue.py) with a name like outgoing_webhooks. The queue worker handling those uses python-requests to make HTTPS requests to clients with a nice user agent like "ZulipWebhook/{zulip_version}", with some reasonable retry policy. I think we can initially do something simple like the outgoing email worker, but before long we'll want to write a simple async server to manage these, since it's an ideal application for async programming.
  • On the frontend, we'll want to do some work on the autocomplete list for @-mentioning users to make it consistent with which users are outgoing webhook bots (currently it's just users subscribed to that stream; we should make it users subscribed to that stream or who are outgoing webhook bots).
  • Potentially, clients can return a JSON-formatted message that they want the outgoing webhook bot to reply with. We'll need to think a bit about how to handle errors if the client returns a malformed response, but probably the programming convenience of this model is worth it.

The actual message format can be similar to the format of messages going into Zulip, e.g.:

content
realm_id
realm_domain
message_type="stream" # will use a different type for private messages (and replace `stream` field).
sender_id
sender_email
sender_full_name
stream_id
stream_name
subject
timestamp
token # we need something the bot can use to authenticate to the target server.
type="message" # in case we want to support other notification event types in the future.

Though it might be simplest to implement that using the same format we use in the send_event() call for a message object...

For implementing this, I'd do this things in roughly this order:

  • start with adding the new bot type, adding code to the auth decorators preventing it from doing anything using the Zulip API, and add tests verifying that code works correctly (we have something similar for incoming webhook bots).
  • then, do the do_send_messages/bugdown changes needed to put events into an event queue
  • then, build queue worker system to deliver messages
  • then fix the frontend autocomplete issues
  • then, add a nice UI (probably a tab in the "create bot" flow?) for creating these
  • then, add some nice formatting in the Zulip administration page explaining what bots exist
  • then, work on adding more features like the ability to set alert_words or have the bot get everything on a specific stream
  • eventually build a Tornado service to make sending outgoing webhook messages highly efficient.
@rahuldeve
Contributor

@timabbott Will it work in private messages if we bots ?

@timabbott
Member

I think it probably makes sense to have PMs sent to the bot be delivered by the webhook system, yeah.

@rahuldeve
Contributor

I meant in the case of a PM between two users. Will one of them be able to trigger an outbound webhook by @-mentioning a bot ? (The message is sent to the other user and not the bot)

@timabbott
Member

Oops, sorry for the slow reply! The answer is basically no, I think we shouldn't support that.

@rahuldeve
Contributor

Hmm, how does that sort of inheritance work from a database table perspective? Will these new bots show up in the UserProfile.objects.filter() queries or not?

It should. Django supports multi table inheritance. On a DB level a seperate table will be created for the OutgoingWebhookBot model with a one on one relationship to the UserProfile table.
https://docs.djangoproject.com/en/1.9/topics/db/models/#multi-table-inheritance

I would have done this how incoming webhook bots are implemented, with the bot_type field, but I'm curious to evaluate whether your approach is better.

I do have a separate bot_type id for the outgoing webhooks in the UserProfile model for identifying them. Unlike incoming webhooks, we have to store some bot specific fields which are not there in UserProfile. I think the easiest way to do this without disturbing the existing table is to create a new one with the extra fields.

Interesting, can you explain why you need something like straight.plugin for this feature?

For now I store all the different bot classes in the zerver/outhooks folder as separate files. When zulip starts I want to load all the classes in memory without having to hardcode anything. Every bot class has a class variable called email. This is used to identify which bot class to instantiate during a trigger.
I did try to make this dynamic loading myself but I couldn't. straight.plugin worked and the code is much more cleaner than what I wrote before.

I had been thinking we should just send the entire message in the webhook (so the integration can handle it how it likes), similar to what e.g. Slack does, but I'm open to playing around with this

I did think about sending either the whole message or just a part of it to the integration when a trigger is activated. Frankly I couldn't decide on one because both seemed fine so I just chose the second option because it seemed more "secure".

I will push my code in a couple of days after refactoring a bit and separating it into proper commits.

@timabbott
Member

Cool. I think we should send the entire message to the hook; there are some integrations that are only possible that way, and I don't think there's a real security issue there.

For bot classes, I guess it's not clear to me why we need bot classes in zerver/outhooks/ at all -- it seems like an outgoing webhook system shouldn't need special code per-third-party-site it's connecting to (the third-party site would be expected to add code to handle the Zulip API format); though I can see a use case for supporting automatically reformatting the messages to match a third-party API; is that latter element what you have in mind for the bot classes? If so, that's potentially worth doing, but probably only makes sense to add once we've gotten the main feature working.

@rahuldeve
Contributor

I think the job of converting a command to an API call should fall on the one developing the feature instead of depending on the third party site. We have the advantage of getting help on developing new integrations from any open source devs but their effort should not be stalled just cause the third party service is not super excited about creating a separate API just for zulip :) And ya reformatting messages do play a part too.

@timabbott
Member

Yeah, OK, I see the argument there. So maybe we should build a "generic" outgoing webhook that's sorta our default for if the third-party site doesn't have a format yet, and sends the full message content (as would be true for an integration built just for Zulip), and then we can also build e.g. a "slack format" outgoing webhook that reformats messages to match Slack's API so one can use any of Slack's outgoing webhook integrations, and we can build a couple more examples (like your isitup bot example), and see what people build from there.

@rahuldeve
Contributor

Sounds good!

@timabbott
Member

Note that #1393 is an implementation effort for this

@timabbott timabbott modified the milestone: Zulip roadmap, Old roadmap Nov 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment