twitter publish: error 32 "Could not authenticate you."

[snarfed]

seen this a few times over the last few days (week) or so, i think just for @gRegorLove and @barryf. (sorry guys!)

example from this log:

D Fetching https://api.twitter.com/1.1/statuses/update.json?status=I+seem+to+have+lost+all+my+saved+searches+in+%40tweetbot+and+the+Twitter+web+app.+I+only+had+a+few+and+can+recreate+but+%C2%AF%5C_%28%E3%83%84%29_%2F%C2%AF
W Error 401, response body: {"errors":[{"code":32,"message":"Could not authenticate you."}]}
W Error 401, response body: {"errors":[{"code":32,"message":"Could not authenticate you."}]}
W Error: {"errors":[{"code":32,"message":"Could not authenticate you."}]} HTTP Error 401: Unauthorized
Traceback (most recent call last):
  File "/base/data/home/apps/s~brid-gy/3.383799630097582262/publish.py", line 195, in _run
    result = self.attempt_single_item(item)
  File "/base/data/home/apps/s~brid-gy/3.383799630097582262/publish.py", line 313, in attempt_single_item
    result = self.source.as_source.create(obj, include_link=not omit_link)
  File "/base/data/home/apps/s~brid-gy/3.383799630097582262/activitystreams/twitter.py", line 405, in create
    return self._create(obj, preview=False, include_link=include_link)
...
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 401: Unauthorized

[snarfed]

on the plus side, i'm doing a happy dance in my seat right now to have a silo bug that's not facebook. (i live a small, sad life. :P)

[barryf]

Hey @snarfed. No need to apologise! -- is there anything I can help with in terms of debugging? I've tried re-authenticating but that still results in the 401.

[snarfed]

@barryf thanks for the offer! i don't know where to start looking yet, and my day job has been pretty busy recently, so it may be a few days before i get to it. definitely feel free to jump in if you want!

[snarfed]

progress: i can reproduce it on https://www.brid.gy/twitter/snarfed_org by trying to publish http://xenon.stanford.edu/~rbarrett/iso.html, but not with the same page and account in dev_appserver. so maybe there's something up with bridgy's twitter app.

[snarfed]

...except we use the same twitter app (key and secret) for localhost and for brid.gy. so i'm kind of stumped. :(

[snarfed]

i've poked at this some more, but no luck. web searches say re-authing as the individual user sometimes fixes it, but not always (including here). they also say it can happen when you change your app-level permissions, or auth as a user with a different x_auth_access_type, like we did in #258...but both of those were a while back, and it's been working since then.

i'm at a wall. anyone have any ideas?

[barryf]

Re-authing seems to be the only recommendation I've seen and, as you say, that doesn't help here.

[snarfed]

next lead i'm going to try: check tweepy's changelog and update if i see anything promising. (or maybe even if i don't.)

[barryf]

Thanks Ryan. If it doesn't look like there's going to be a simple fix I'll happily write my own Twitter syndication code (I'd started down that track anyway but found your implementation was better) and I guess others could also do so. As you say, you've got quite a lot on your plate with your day job (congrats!).

[snarfed]

i upgraded tweepy, but no luck. i'm out of ideas. :/

[snarfed]

another data point: i tried locally with the same access token as an account in prod, and it worked. so they've blacklisted app engine, or something else about the prod environment that i can't find is unhappy.

[snarfed]

posted this on twitter's developer forum:

hi all! my app has been happily posting tweets via /1.1/statuses/update for a year or so. a week ago it started returning HTTP 401 {"errors":[{"code":32,"message":"Could not authenticate you."}]} for all of my users. i hadn't changed any of my relevant code or app settings recently.

here's the particularly odd part: when i try it in my local environment with the same code, twitter app key/secret, and even the exact same user access token key/secret, it works fine. it only fails in production.

i'm on google app engine using tweepy. (tried versions 2.2 and 3.3, the latest. both fail.) could app engine's external facing IPs have been blacklisted or graylisted somehow?

more details in this github issue. any ideas? thanks in advance!

[snarfed]

the plot thickens: a number of people chimed in on that forum post with the same problem.

i've also posted on the GAE forum.

[kylewm]

major respect to the three of you debugging a black box. that's awesome to watch.

[snarfed]

aww thanks! especially to dave loomer in that GAE forum thread, he's diving really deep.

[snarfed]

to summarize the current state of affairs, we think app engine 1.9.20 introduced new URL-escaping behavior into the urlfetch API that sometimes breaks OAuth 1.1 signatures in Twitter API calls. we see this on 1.9.20 instances but not 1.9.19. (i'm guessing it would also happen with other OAuth 1.1 services, e.g. Tumblr.)

a google devrel person pointed us to this app engine issue, which sounds like the same thing. feel free to star it!

[snarfed]

app engine fixed their bug; bridgy publish can tweet again! yay. glad i can close this.

[barryf]

Wow. True needle-in-a-haystack debugging. Thanks very much for persevering with this. I'm happily Bridgy publishing again.