Twitter Stream re-connect best practices

[dubielt1]

I have a project that will consume a long-running twitter stream, and the documentation doesn't mention what will happen when a disconnect occurs. Looking at the code, a requests exception might get caught, at which point python-twitter would raise its own exception?

I want to be sure that I take the best approach for guaranteeing a re-connect after being disconnected.

I apologize if I overlooked anything in the documentation, or if this behaviour is inherent to requests in some way.

[jeremylow]

No apologies necessary! I didn't know what would happen until you asked.

You can set the requests.Session timeout parameter with something like:

api = twitter.Api(
	CONSUMER_KEY,
	CONSUMER_SECRET,
	ACCESS_TOKEN,
	ACCESS_SECRET,
	tweet_mode="extended",
	timeout=3.0  # three second timeout
)

at which point, if something goes wrong (your connection goes down) you'll get a ConnectionError, which you could catch. If your connection comes back up before you disconnect with a ConnectionError, you might see some other errors. In testing, I got the following:

IncompleteRead,
ChunkedEncodingError,
ProtocolError

etc. I'm not sure that covers all of them, so you might want to implement some logging function and check periodically.

If you don't set a timeout when instantiating the api, then the stream will just hang indefinitely.

For other things, like Twitter disconnecting you, you'll get a TwitterError. They provide pretty detailed documentation on this here: https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data

Hope that helps!

[dubielt1]

Exactly what I was looking for, thank you