Run IRC bot for salt interactions

[jdm]

Example of the single command that saltbot understands right now:

14:22 <jdm> disk usage
14:22 <saltbot> servo-linux1: 68%
14:22 <saltbot> servo-linux5: 47%
14:22 <saltbot> servo-linux4: 70%
14:22 <saltbot> servo-linux6: 37%
14:22 <saltbot> servo-linux3: 68%
14:22 <saltbot> servo-mac2: 75%
14:22 <saltbot> servo-mac3: 35%
14:22 <saltbot> servo-mac8: 31%
14:22 <saltbot> servo-mac6: 32%
14:22 <saltbot> servo-mac7: 9%
14:22 <saltbot> servo-mac4: 37%
14:22 <saltbot> servo-mac5: 32%
14:22 <saltbot> servo-mac1: 16%
14:22 <saltbot> servo-linux2: 63%

Since salt commands can only be run as root (or they can't write to logs in /var/logs), I couldn't figure out how to run this bot as non-root. The code for the bot is at https://github.com/jdm/saltbot. @aneeshusa, what are your thoughts on this?

---

This change is 

[aneeshusa]

@jdm, thanks for working on this! I'm always glad to see progress on chatops for servo.
A couple notes:

• Instead of using cmd.run 'df -h', use Salt's built in disk.usage execution function, which will give you structured data, and should work on Windows too. Might be nice to also have the IRC interface reflect this and be disk.usage instead of disk usage (or accept both possibly).
• It looks like the current code is in Javascript; I'd prefer to use Salt's Python client API, which gives you structured data and is a bit easier to use IMO. In my experience trying to run NodeJS in production causes operational sadness. Otherwise, you'll probably be interested in using the JSON outputter, which will cause Salt to output (machine-readable) JSON.
• Thanks for trying to run this as non-root; I looked into this previously as part of #657 and there's currently not a great way to do this. A few options:
  • I'm OK with running this as root, but we'll need to be careful about what access we provide via IRC.
  • Another (more involved) option is setting up their CherryPy REST API, and then hitting it directly or using the Pepper Python Client to use that API. This uses Salt's pluggable External Auth system, which will allow us to create a Salt user just for the IRC bot and restrict which functions and arguments it can use. If we do this, we can run just the CherryPy API on the Salt master, set up TLS, and then run the IRC bot on a separate machine for added safety.
• If you decide to keep invoking Salt via the CLI binary, then you should run one Salt command with a target of all builders instead of invoking Salt once for each builder; Salt should stream output for you as results come back. salt-key -L is useful to figure out the set of current minions, instead of hardcoding the number of Linux/macOS minions.
• Some form of rate-limiting would be good in the bot to avoid being DOSed. Similarly, we should probably check that incoming messages are to the bot.
• I can see this being extended to not-fully-Salt related things (e.g. which saltfs PRs are outstanding/yet-to-be-deployed? This actually is a GitHub call), so maybe pick a different name while it's easy.

Hooking up IRC to Salt is a bit scary, so I'll try to take a deeper look later on.