Flask server for interacting with the contrib bots through outgoing webhooks. #4864

vabs22 · 2017-05-22T16:24:35Z

No description provided.

vabs22 · 2017-05-22T21:20:39Z

Things to keep in mind:

The flask server needs to be run inside the vagrant machine.
It supports followup bot for now, the followup bot config details need to be added in the form specified in comments, in bot_config dict().
To make it work for other bots, the base_url in populate_db changes need to be updated to http://127.0.0.1:5002/bots/<botname> and the corresponding config of bot needs to be added into bot_config dict().

Edit: updated server address

robot-dreams

(Edit): Oops, I left a comment in the wrong window!

timabbott · 2017-05-24T04:19:59Z

zilencer/management/commands/populate_db.py

+            create_users(zulip_realm, zulip_outgoing_bots, bot_type=UserProfile.OUTGOING_WEBHOOK_BOT)
+            Service.objects.create(name="test",
+                                   user_profile=get_user_profile_by_email("outgoing-webhook@zulip.com"),
+                                   base_url="http://0.0.0.0:5002/bots/followup",


Why is this 0.0.0.0:5002? Seems like maybe we want it to be 127.0.0.1:5002 instead.

timabbott · 2017-05-29T23:05:42Z

I merged a tweaked version of the outgoing webhook commit, so the basic outgoing webhook user should be available in the test suite now.

I think we should do a bit of cleanup of the flask server system next. I think the main thing that this needs is a configuration system that we can use to move the API keys and other settings out of the flask server code and into a config file.

I would suggest using a ConfigParser / zuliprc file style approach, similar to what our other bots do, though it might make sense to have a variation in the approach to support having multiple bots defined in the same file?

robot-dreams

Thanks for merging the test commit, @timabbott ! @vabs22 , I think you can rebase so that this PR only has the Flask commit.

I left a few comments inline; otherwise, I agree with working on the configuration next. Please feel free to PM me if you want to discuss the format for configuration files, or if you need help with getting the ConfigParser to work.

robot-dreams · 2017-05-30T09:54:55Z

contrib_bots/contrib_server.py

+                                   client=restricted_client,
+                                   state_handler=state_handler)
+
+    # return json_success("Success!")


Should this be "Success!" or json_success("Success!")? Either way, can you please choose one and then remove the comment?

robot-dreams · 2017-05-30T09:55:59Z

contrib_bots/contrib_server.py

+    path = "bots/" + str(bot) + "/" + str(bot) + ".py"
+    bots_lib_module[bot] = get_lib_module(path)
+
+class StateHandler(object):


I think StateHandler has been extracted and should be available now, without needing to redefine it here.

robot-dreams · 2017-05-30T09:57:07Z

contrib_bots/contrib_server.py

+def test(bot):
+    # type: (str) -> str
+    # if bot not in available_bots:
+    #     return "requested bot service not supported"


In general, it's helpful for error messages to include more information about the error. In this case, for example, I would prefer something like this:

return "requested bot service {} not supported".format(bot)

robot-dreams · 2017-05-30T09:57:44Z

contrib_bots/contrib_server.py

+    return ""
+
+@app.route('/bots/<bot>', methods=['POST'])
+def test(bot):


Nit: Is test the name you want here?

robot-dreams · 2017-05-30T09:57:59Z

contrib_bots/contrib_server.py

+@app.route('/bots/<bot>', methods=['POST'])
+def test(bot):
+    # type: (str) -> str
+    # if bot not in available_bots:


Can you please uncomment or remove?

robot-dreams · 2017-05-30T09:59:26Z

contrib_bots/contrib_server.py

+    state_handler = StateHandler()
+
+    event = json.loads(request.data)
+    message_handler.handle_message(message=event["message"],


Do you think it would be helpful to add a comment about the expected format of the request somewhere (i.e. the fact that "message" is the relevant key, as well as the expected format of event["message"])?

Yeah it would be helpful to add the comment. Currently we are sending the data directly but later on I think we would be passing it through an interface designed for this server. That might be the best place for it. Do you want me to add a TODO comment inside DoRestCall?

That sounds good for now.

vabs22 · 2017-05-30T17:58:32Z

@timabbott @robot-dreams Thanks for you reviews!. I have incorporated the comments and updated it.
I also think it would be reasonable to have a single file containing the configurations of all bots. I have completed an intial version for the same and added a new commit which generates a single config file for all active bots. The sample format of the config file is as above:

For downloading the above file I have added a download button similar to existing one. It can be seen here:

I have also updated the flask server to read the configuration file and then populate the bots_config dict using SafeConfigParser. Please share your thoughts.

robot-dreams

Thanks, this is a nice approach to configuration! I left a few comments inline; please take a look.

robot-dreams · 2017-06-04T04:38:23Z

static/js/settings_bots.js

@@ -56,6 +56,14 @@ exports.generate_zuliprc_content = function (email, api_key) {
           "\n";
 };

+exports.generate_zuliprc_content_for_single_file = function (email, api_key) {
+    return "[" + email.substring(0, email.indexOf("-bot@zulip.localhost")) + "]" +


Will this always be @zulip.localhost? Would it make sense to use email.indexOf("-bot@") instead?

Also, would this code be more readable with a separate bot_name_from_email function?

I agree with both of your points.

robot-dreams · 2017-06-04T04:42:31Z

static/templates/settings/bot-settings.handlebars

@@ -59,6 +59,13 @@

        <div class="form-horizontal" id="api_key_button_box">
            <div class="input-group side-padded-container">
+                <div>
+                    <span>Download single zuliprc of all active bots</span>


Could the zuliprc name cause confusion because it's also used for other kinds of configuration files?

If so, would a name like zulipbotrc or flaskbotrc or (you can probably come up with something better) make sense?

yeah flaskbotrc would make more sense to the users.

robot-dreams · 2017-06-04T04:49:37Z

contrib_bots/contrib_server.py

+app = Flask(__name__)
+
+@app.route('/')
+def main():


Just checking, is it necessary to define this route / function?

If so, can it be called something besides main()?
If not, can it be removed?

I had added this initially for testing, so it can be removed now. I should have removed it much earlier. My bad.

robot-dreams · 2017-06-04T04:50:08Z

contrib_bots/contrib_server.py

+    state_handler = StateHandler()
+
+    event = json.loads(request.data)
+    message_handler.handle_message(message=event["message"],


That sounds good for now.

robot-dreams · 2017-06-04T04:58:39Z

contrib_bots/contrib_server.py

+
+from zulip import Client
+
+config_file_path = sys.argv[1]


Referencing sys.argv[1] at the top level like this seems like it could lead to confusion or problems—especially if this module gets imported somewhere else.

While this is pretty clearly a standalone app and probably won't ever be imported, I still think it's important to consistently use good habits across the codebase.

Instead of the current approach, do you think it would make sense to define a load_config function that takes a file_path as a parameter, and then call it from the if __name__ == "__main__" block?

Yeah it would be much better to shift these code lines into separate functions and call them from if __name__ == "__main__" block.

robot-dreams · 2017-06-04T04:59:14Z

contrib_bots/contrib_server.py

+                    site=bots_config[bot]["site"])
+    restricted_client = BotHandlerApi(client)
+    message_handler = bots_lib_module[bot].handler_class()
+    state_handler = StateHandler()


Can you please add a TODO to handle stateful bots (e.g. the Tic Tac Toe bot)?

robot-dreams · 2017-06-04T05:01:04Z

static/templates/settings/bot-settings.handlebars

@@ -59,6 +59,13 @@

        <div class="form-horizontal" id="api_key_button_box">
            <div class="input-group side-padded-container">
+                <div>
+                    <span>Download single zuliprc of all active bots</span>


Would it be helpful to say what this is for (e.g. Download config of all active bots for Flask bot server or something)?

yeah, I agree.

robot-dreams

Looks good to me. Thanks for addressing all of my comments! I can't wait to see this in action.

timabbott · 2017-06-13T05:30:29Z

@vabs22 I merged the first commit to help make it easy for other folks to collaborate on the flask system, congrats on getting that done! I noticed a few things we'll want to fix in the near future:

python api/flask_bot_server.py doesn't work; you have to run it directly from api/. Since with the pip installer, this will be available from just about any directory, we definitely will want to work on this element.
Structurally, flask_api_server.py might belong in the api/zulip directory; I'm not sure, and possibly also should be executable. I didn't move it when merging because its path handling is fragile (see above) and I didn't want to debug potential issues there before making it possible for multiple folks to work on this code easily.
We should use optparse/argparse to make --help work, so it's easy to figure it out.
We might want to rename it (at least when installed) to have a reasonable name in PATH. I.e. I think we kinda want the command after a pip install to be zulip-bot-server ~/flaskbotrc.
The error message if your flaskbotrc file has as invalid bot name is super confusing. We should be really clear that the problem is that there is no bot with name "test" if your bot is called "test-bot", and that the most likely solution is to edit your flaskbotrc file to fix the name of the bot to match the bot you want to run.

@eeshangarg, as the maintainer of the pip package, can you collaborate with @vabs22 on these follow-up tasks (which I think we should try to get done this week, since they should mostly be quick.).

More medium-term, we will want to work on these:

Add a basic test suite for the Flask system so we can change it with minimal fear.
supervisord configuration for running the flask bot persistently that's super easy to setup (e.g. install-zulip-flask-supervisord <path_to_flaskbotrc>
We should work on writing trivial heroku instructions for using this.

@vabs22 can you and @robot-dreams open issues for all of the above follow-up tasks that aren't going to be done this week? I imagine those will be your projects for the next week or two.

For the JS changes, I opened #5355 for a related cleanup we'll want to do, which would be reasonable for you to work on @vabs22. The remaining patch in this PR I can merge once we fix a couple small things:

"Download config of all active bots for Flask bot server" needs to be tagged for translation
"Download config of all active bots for Flask bot server" is confusing. I think we want to name the "Flask bot server" something more suggestive, e.g. "Zulip Botserver" (I don't think our users should care that's based on Flask). Also, it should probably only download the outgoing webhook bots, since e.g. incoming webhooks aren't useful to put in this flask server. So maybe something like "Download config of all active outgoing webhooks in Zulip Botserver format."

vabs22 · 2017-06-16T11:00:55Z

I have updated this PR, the UI looks like:

Once the outgoing webhook UI gets merged, I will update the tests.

showell · 2017-06-19T20:09:54Z

@vabs22 Can you clean up the commit message here? The title line should be no longer than 65 characters, and then you can have an empty line and more text to elaborate as needed.

Flaskbotrc is a file containing config of all active outgoing webhook bots. It is used to provide configuration of all active outgoing webhook bots to zulip-bot-server.

vabs22 · 2017-06-20T13:22:39Z

@showell I have updated the commit message and the tests. Please take a look.

showell · 2017-06-20T13:43:54Z

Merged! Thanks, @vabs22!

zulipbot added the needs review label May 22, 2017

vabs22 force-pushed the flaskserverbots branch from 4f264df to 03e801e Compare May 22, 2017 21:00

robot-dreams suggested changes May 23, 2017

View reviewed changes

zulipbot added needs more work and removed needs review labels May 23, 2017

vabs22 force-pushed the flaskserverbots branch from 03e801e to 72d8295 Compare May 23, 2017 20:08

zulipbot added needs review and removed needs more work labels May 23, 2017

robot-dreams mentioned this pull request May 23, 2017

Add support for embedded bots #4954

Closed

7 tasks

timabbott reviewed May 24, 2017

View reviewed changes

zulipbot added needs more work and removed needs review labels May 24, 2017

vabs22 force-pushed the flaskserverbots branch from 72d8295 to 0276cdc Compare May 24, 2017 15:30

zulipbot added needs review and removed needs more work labels May 24, 2017

vabs22 force-pushed the flaskserverbots branch 3 times, most recently from c92248e to 7643b6d Compare May 28, 2017 19:40

vabs22 changed the title ~~(WIP) Flask server for interacting with the contrib bots through outgoing webhooks~~ Flask server for interacting with the contrib bots through outgoing webhooks. May 28, 2017

robot-dreams reviewed May 30, 2017

View reviewed changes

zulipbot added needs more work and removed needs review labels May 30, 2017

vabs22 force-pushed the flaskserverbots branch from 7643b6d to e830f78 Compare May 30, 2017 17:46

zulipbot added needs review and removed needs more work labels May 30, 2017

vabs22 force-pushed the flaskserverbots branch 2 times, most recently from 5a720dd to 39b5e16 Compare May 30, 2017 20:09

vabs22 force-pushed the flaskserverbots branch from 39b5e16 to 9e0eb6a Compare May 31, 2017 07:05

robot-dreams reviewed Jun 4, 2017

View reviewed changes

zulipbot added needs more work and removed needs review labels Jun 4, 2017

vabs22 force-pushed the flaskserverbots branch from 9e0eb6a to 13f7eae Compare June 5, 2017 18:37

zulipbot added needs review and removed needs more work labels Jun 5, 2017

vabs22 force-pushed the flaskserverbots branch 3 times, most recently from d12e0ad to f663269 Compare June 6, 2017 16:24

robot-dreams approved these changes Jun 12, 2017

View reviewed changes

zulipbot added needs more work and removed needs review labels Jun 12, 2017

roberthoenig mentioned this pull request Jun 14, 2017

docs: Add flask server documentation. #5322

Merged

vabs22 force-pushed the flaskserverbots branch from f663269 to 550be01 Compare June 16, 2017 10:44

zulipbot added needs review and removed needs more work labels Jun 16, 2017

vabs22 force-pushed the flaskserverbots branch from 550be01 to 8dc74ec Compare June 16, 2017 16:04

vabs22 force-pushed the flaskserverbots branch from 8dc74ec to ae9ef5f Compare June 20, 2017 11:18

Add support for generating and downloading flaskbotrc.

ec55760

Flaskbotrc is a file containing config of all active outgoing webhook bots. It is used to provide configuration of all active outgoing webhook bots to zulip-bot-server.

vabs22 force-pushed the flaskserverbots branch from ae9ef5f to ec55760 Compare June 20, 2017 12:38

showell merged commit 9a6e326 into zulip:master Jun 20, 2017

zulipbot removed the needs review label Jun 20, 2017

Flask server for interacting with the contrib bots through outgoing webhooks. #4864

Flask server for interacting with the contrib bots through outgoing webhooks. #4864

Conversation

vabs22 commented May 22, 2017

vabs22 commented May 22, 2017 • edited Loading

robot-dreams left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timabbott commented May 29, 2017

robot-dreams left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vabs22 commented May 30, 2017

robot-dreams left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robot-dreams left a comment • edited Loading

Choose a reason for hiding this comment

timabbott commented Jun 13, 2017 • edited Loading

vabs22 commented Jun 16, 2017

showell commented Jun 19, 2017

vabs22 commented Jun 20, 2017

showell commented Jun 20, 2017

vabs22 commented May 22, 2017 •

edited

Loading

robot-dreams left a comment •

edited

Loading

robot-dreams left a comment •

edited

Loading

timabbott commented Jun 13, 2017 •

edited

Loading