Add Twitch Connector

[FabioRosado]

Description

This is the initial commit for the Twitch connector, there is still a lot to be done like writing the tests and create the documentation for it. I'm raising this draft PR to get some feedback on the word done and if I can improve things a bit more.

This connector is pretty big because I'm using a lot of things. Twitch has different ways to do stuff so we are using webhooks to get 3 events, then calling the API to update channel titles and websockets to listen to the chat.

I thought about what other bots do and what could be cool to include and that's the reason this connector grew so much haha

Things still to be done

• Update documentation
• Write tests for the whole connector
• Test most of the features on Twitch (did some but still need further testing)
• Write the skill-twitch to handle different events and actions
• Figure out why websockets throw exception when exiting opsdroid
• Add ssl support to connect to websockets - still need to figure out how to do that.
• Check if every logger contains _()
• Figure out and implement hub.secret check when handling webhook response - prevent folks from sending data that wasn't sent by Twitch.
• Connect to the chat service only when streamer went online.
• Handle aiohttp status code != 200. Currently I'm assuming everything works fine - that is BAAAAAD

Notes

• `config['forward-url'] - Added this so we can still run opsdroid on localhost and use ngrok or serveo forwarding urls as a callback when subscribing to webhooks.
• token expires after 24 hours so we will refresh token when it expires and will re-subscribe to webhooks
• twitch.json - saved data on appdir data to keep the oauth token and refresh token saved so we can get/set
• We should really only connect to the twitch irc whenever a streamer goes live, it's unlikely that we will do anything with the chat until we are online. I need to check if the webhook will still receive data from Twitch after the oauth token expired (but the webhook lease is still going), that way we can keep the 'refresh' token code on the websockets connect method

Every time we send something through websocket the command is very similar, should we just create a helper method to send messages to the websocket? Will this improve readability?

Status

READY | UNDER DEVELOPMENT | ON HOLD

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Documentation (fix or adds documentation)

How Has This Been Tested?

• Test A
• Test B

Checklist:

• I have performed a self-review of my own code
• I have made corresponding changes to the documentation (if applicable)
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[codecov]

Codecov Report

Merging #1497 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##           master    #1497    +/-   ##
========================================
  Coverage   99.88%   99.89%            
========================================
  Files          54       56     +2     
  Lines        3395     3686   +291     
========================================
+ Hits         3391     3682   +291     
  Misses          4        4            

Impacted Files	Coverage Δ
opsdroid/connector/twitch/__init__.py	100.00% <100.00%> (ø)
opsdroid/connector/twitch/events.py	100.00% <100.00%> (ø)
opsdroid/const.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 28eb9af...6aa2364. Read the comment docs.

[jacobtomlinson]

This is really nice! Especially the docstrings. I've made a bunch of small comments.

I'm keen to think about testing this a little differently. Generally in the past we've always done small unit tests which test each function individually and mock everything else completely. This is very rigid and has caused a right headache when refactoring or changing things later.

I propose we move to a different model where our tests start up a dummy websocket server, mock out the API URL constants to point to our local server and run things end to end through opsdroid with some test skills.

I'm about to do this myself in the GitHub connector refactor, so perhaps you want to hold off writing tests until I've done mine so you have an example to work from?

[FabioRosado]

Hello Jacob I've resolved all of the comments that you've made and did some testing of the connector on my end on stream as well. Did some more refactoring and decided to add a always-listening flag to the config, if this flag is set opsdroid will keep trying to connect and reconnect to the twitch server.

The always-listening flag defaults to false so the default behavior of the connector is to listen to the webhook and only start listening to the chat whenever a broadcaster went live. This seemed like the best approach since the chat will be empty when the broadcaster isn't there and twitch keps closing the websocket connection.

I've been thinking about the webhooks. Currently Twitch asks you to set a lease for the subscription to expire. I assume that Twitch will keep sending events even if our oauth expired (but need to test this out on my end).

I'm thinking to somehow add a crontab to run a method that will trigger a token refresh. Should this be done in the connector or on a skill? 🤔

[jacobtomlinson]

I would do it in the listen. You are looping there right? So put a check in that says something like if time_since_last_check > 1hr: refresh().

[FabioRosado]

Sorry for the late reply, I've refactored the code a bit and we are not really using the loop to listen the websockets since I made the decision to listen for the "is live" event sent through the webhook and then connect to the chat service

[FabioRosado]

I've been livetesting the connector and the bot on stream for the past 3 streams and I'm trying to squash some annoying bugs. Twitch seems to close the websockets more often when I'm live (even if opsdroid is open on a digital ocean droplet).

I tried to do the return on the reconnect() but that made the websockets hang and wasn't able to reconnect to the websocket server. Also when it hanged I couldn't shut the bot with CTRL+C for some reason 🤔

[jacobtomlinson]

You are looking at this from the developer end of the telescope because you've been writing it, it's only natural. But I recommend you try and take a step back and think about this experience from the user's perspective.

Try not to give them unnecessary detail about how the connector works and instead tell them about how they can use it.

Also try and include some small skill examples if you can.

[FabioRosado]

Very good points haha will change things around and add some examples 😄

[FabioRosado]

Btw after using digital ocean for a few days I've noticed that for some reason ubuntu keeps killing opsdroid, so I'm afraid websockets is giving us a memory leak of some sort. Not sure how to fix now but might give it a go and try aiohttp websockets support, maybe its a library things - I think slack was using the same websockets lib? 🤔

[jacobtomlinson]

Yeah the old slack connector had a memory leak. It seemed to happen when the socket was closed from the remote end and a reconnect happened. Not sure why.

We ended up switching to the official Slack python library when they added asyncio support and we haven't seen the problem since. So might be worth looking at how they implemented the websockets support. Looks like they use aiohttp ws_connect.

[FabioRosado]

That's what I thought as well, since it seems to be happening everytime the websocket gets closed by Twitch, so I thought it might be an issue with how I'm trying to reconnect or it's a websocket problem.

Thank you for pointing the slack library so I can see how they implement the reconnect will play around and try to solve the issue 👍

[FabioRosado]

Apologies for all of these pushes, I've tested things out and the connector seems to work okay. Still going to test the websocket connect tomorrow on stream to see if I fixed all the issues. It will probably take me a while to write the tests for it and update the docs but I'm happy with the code so far.

Also still need to test the renewal logic on a long session of opsdroid.

[FabioRosado]

Well I thought the issue with the docs(docstring) was because I split up that line, but that wasn't it. Can't really understand what
/home/runner/work/opsdroid/opsdroid/opsdroid/connector/twitch/__init__.py:docstring of opsdroid.connector.twitch.ConnectorTwitch.twitch_webhook_handler:10:Inline interpreted text or phrase reference start-string without end-string.

this means. I thought it was a line that was missing punctuation but doesn't seem like it 🤔

Edit: Yeah... it wasn't the command, so I have no idea haha

[jacobtomlinson]

I have no idea why things are failing here.

I've made a couple of suggestions just to see if they fix it. But I'm wildly stabbing in the dark.

One thing that comes to mind is that code blocks in rst use double backticks and most of your docstrings use single backticks. However it doesn't seem to be complaining in other places. You may just want to fix it anyway.

In rst you highlight ``code`` like this.

[FabioRosado]

Got to say, adding two backticks and a space its a very annoying thing haha but it's fine 😛 I just need to get used to that, I've added two backticks on all of them and also added the space, lets see if that will fix it.

[Cadair]

in rst single backtick is a (or can be) a cross-reference.

[FabioRosado]

Ah okay makes sense, and know that I know why it was complaining it makes sense why it was saying start-end issues. Good to see that the thing passes now haha

[Cadair]

This looks really awesome!! 🚀

I have just reviewed the event handling stuff.

[jacobtomlinson]

So glad to see this go in. It has been a lot of work. Many thanks @FabioRosado.