New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

twitter streaming backend is broken #57

Closed
snarfed opened this Issue Feb 7, 2014 · 6 comments

Comments

Projects
None yet
1 participant
@snarfed
Owner

snarfed commented Feb 7, 2014

pretty bad leak. looks like it was introduced around feb 3 morning:

chart

the only damage is that the backend restarts when it hits the soft limit, so not too big a deal.

@snarfed snarfed added now and removed later labels Feb 14, 2014

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Feb 14, 2014

Owner

the root cause here turned out to be pretty bad. user stream connections are failing with 401 Unauthorized. not sure why yet, except that we have 53 twitter accounts, which may be over the limit for individual user streams from a single IP (or whatever heuristic twitter uses).

same problem happens with a different twitter app key, so it's not that bridgy got blacklisted or something.

Owner

snarfed commented Feb 14, 2014

the root cause here turned out to be pretty bad. user stream connections are failing with 401 Unauthorized. not sure why yet, except that we have 53 twitter accounts, which may be over the limit for individual user streams from a single IP (or whatever heuristic twitter uses).

same problem happens with a different twitter app key, so it's not that bridgy got blacklisted or something.

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Feb 14, 2014

Owner

narrowed it down to snarfed/oauth-dropins@5a21b06. works with that change reverted.

Owner

snarfed commented Feb 14, 2014

narrowed it down to snarfed/oauth-dropins@5a21b06. works with that change reverted.

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Feb 15, 2014

Owner

grr. it's now working on my local machine, but not deployed.

Owner

snarfed commented Feb 15, 2014

grr. it's now working on my local machine, but not deployed.

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Feb 15, 2014

Owner

things i tried that didn't fix it:

  • confirmed i'm using https://userstream.twitter.com/1.1/user.json (note the https and 1.1). also tried http and 2 (which uses API v1). no luck.
  • checked that twitter's server time and my server time are in close enough sync, ie within seconds. evidently oauth 1 signature validation can depend on this.
  • waited overnight to let any temporary blocks expire (who knows if that's long enough), then tried reconnecting just a few users, instead of all 53.
  • rolled back tweepy to before tweepy/tweepy@9f06a21 and tweepy/tweepy@d1c722c, which changed their user stream code.
  • tried all sorts of changes to how i import modules. symlinks, sys.path and sys.module munging, etc. i could consistently reproduce the 401 for a single user, and then fix it again, with different variations of this. no clue why. i'd dismiss it as FUD except it consistently worked with some variations (locally, not deployed) and not others.

at this point, i suspect twitter has a threshold of 50 user stream per IP. over that, they start blocking IPs. thanks to @bear for suggesting this in IRC. i'll probably switch to scraping for favorites. :/

Owner

snarfed commented Feb 15, 2014

things i tried that didn't fix it:

  • confirmed i'm using https://userstream.twitter.com/1.1/user.json (note the https and 1.1). also tried http and 2 (which uses API v1). no luck.
  • checked that twitter's server time and my server time are in close enough sync, ie within seconds. evidently oauth 1 signature validation can depend on this.
  • waited overnight to let any temporary blocks expire (who knows if that's long enough), then tried reconnecting just a few users, instead of all 53.
  • rolled back tweepy to before tweepy/tweepy@9f06a21 and tweepy/tweepy@d1c722c, which changed their user stream code.
  • tried all sorts of changes to how i import modules. symlinks, sys.path and sys.module munging, etc. i could consistently reproduce the 401 for a single user, and then fix it again, with different variations of this. no clue why. i'd dismiss it as FUD except it consistently worked with some variations (locally, not deployed) and not others.

at this point, i suspect twitter has a threshold of 50 user stream per IP. over that, they start blocking IPs. thanks to @bear for suggesting this in IRC. i'll probably switch to scraping for favorites. :/

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Feb 15, 2014

Owner

here's the timeline, just for my own info later:

so, the 401s started immediately when i switched to api v1.1. i wasn't at 50 users yet, but the v1.1 limits may be different from v1. also, i assume twitter finally turned off the old v1 endpoint (/2/user.json) around feb 3 11:10am PST, but it doesn't really matter here.

Owner

snarfed commented Feb 15, 2014

here's the timeline, just for my own info later:

so, the 401s started immediately when i switched to api v1.1. i wasn't at 50 users yet, but the v1.1 limits may be different from v1. also, i assume twitter finally turned off the old v1 endpoint (/2/user.json) around feb 3 11:10am PST, but it doesn't really matter here.

@snarfed

This comment has been minimized.

Show comment
Hide comment
@snarfed

snarfed Feb 16, 2014

Owner

switched to scraping HTML. main changes were snarfed/activitystreams@d37ce67160f8c2dde9de15c5b21071f2ce283f2c and snarfed/activitystreams@ca00d12c7536e82356d1b3fe6eb993ad099d2002.

Owner

snarfed commented Feb 16, 2014

switched to scraping HTML. main changes were snarfed/activitystreams@d37ce67160f8c2dde9de15c5b21071f2ce283f2c and snarfed/activitystreams@ca00d12c7536e82356d1b3fe6eb993ad099d2002.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment