/acd_cli

New issue

Error refreshing authentication token (Appspot server over quota) #147

Closed
amaxcz opened this Issue Sep 20, 2015 · 47 comments

Comments

Assignees
No one assigned
Labels
AWS apocalypse oauth
Projects
None yet
Milestone
No milestone
@amaxcz

amaxcz commented Sep 20, 2015

Now ACD have some troubles from my side.
This application is temporarily over its serving quota. Please try again later.

root@backup:~# acdcli sync
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/oauth.py", line 17, in create_handler
    return LocalOAuthHandler(path)
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/oauth.py", line 218, in __init__
    self.load_client_data()
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/oauth.py", line 225, in load_client_data
    with open(cdp) as cd:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/acd_cli/client_data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/acdcli", line 9, in <module>
    load_entry_point('acdcli==0.3.0a6', 'console_scripts', 'acdcli')()
  File "/usr/local/bin/acd_cli.py", line 1239, in main
    acd_client = client.ACDClient(CACHE_PATH)
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/client.py", line 31, in __init__
    self.handler = oauth.create_handler(path)
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/oauth.py", line 19, in create_handler
    return AppspotOAuthHandler(path)
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/oauth.py", line 131, in __init__
    self.load_oauth_data()
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/oauth.py", line 75, in load_oauth_data
    self.get_auth_token(reload=False)
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/oauth.py", line 90, in get_auth_token
    self.refresh_auth_token()
  File "/usr/local/lib/python3.4/dist-packages/acdcli/api/oauth.py", line 170, in refresh_auth_token
    'Error refreshing authentication token: %s' % response.text)
acdcli.api.common.RequestError: RequestError: 1004, Error refreshing authentication token:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>503 Over Quota</title>
<style><!--
body {font-family: arial,sans-serif}
div.nav {margin-top: 1ex}
div.nav A {font-size: 10pt; font-family: arial,sans-serif}
span.nav {font-size: 10pt; font-family: arial,sans-serif; font-weight: bold}
div.nav A,span.big {font-size: 12pt; color: #0000cc}
div.nav A {font-size: 10pt; color: black}
A.l:link {color: #6f6f6f}
A.u:link {color: green}
//--></style>

</head>
<body text=#000000 bgcolor=#ffffff>
<table border=0 cellpadding=2 cellspacing=0 width=100%>
<tr><td bgcolor=#3366cc><font face=arial,sans-serif color=#ffffff><b>Error</b></td></tr>
<tr><td>&nbsp;</td></tr></table>
<blockquote>
<H1>Over Quota</H1>
This application is temporarily over its serving quota.  Please try again later.

<p>
</blockquote>
<table width=100% cellpadding=0 cellspacing=0><tr><td bgcolor=#3366cc><img alt="" width=1 height=4></td></tr></table>
</body></html>
@judhat2

judhat2 commented Sep 20, 2015

Amazon Cloud Drive is currently having some problems. I am getting the same message and can't access their website.
@msh100
Contributor

msh100 commented Sep 20, 2015

+1 I called them earlier. They're investigating

Sent from my iPhone

On 20 Sep 2015, at 15:16, judhat2 <notifications@github.commailto:notifications@github.com> wrote:

Amazon Cloud Drive is currently having some problems. I am getting the same message and can't access their website.

Reply to this email directly or view it on GitHubhttps://github.com/yadayada/acd_cli/issues/147#issuecomment-141791051.
@amaxcz

amaxcz commented Sep 20, 2015

ofc, its a good reason to make a fix acdcli now :)
@msh100
Contributor

msh100 commented Sep 20, 2015

You can't fix acdcli if Amazon is down...

I get 500 errors btw, not the same as you.

Sent from my iPhone

On 20 Sep 2015, at 15:17, amaxcz <notifications@github.commailto:notifications@github.com> wrote:

ofc, its a good reason to make a fix acdcli now :)

Reply to this email directly or view it on GitHubhttps://github.com/yadayada/acd_cli/issues/147#issuecomment-141791499.
@yadayada
Owner

yadayada commented Sep 20, 2015

ACD was down for a few hours. That lead to a massive spike in requests to the App Engine OAuth server. So, now it's over quota for today and you'll have to wait 17 hours for the quota to be reset.

screenshot-app engine dashboard - acd-cli-oa - google chrome-1

yadayada added the oauth label Sep 20, 2015

yadayada referenced this issue Sep 20, 2015

Closed

Error during sync "This application is temporarily over its serving quota. Please try again later." #148

yadayada changed the title from please track ACD failures in proper way and return right message and error code to Error refreshing authentication token (Appspot server over quota) Sep 20, 2015

@jbuchbinder

jbuchbinder commented Sep 20, 2015

Any chance of checking in the appspot code (since it's not accessible when the server is over quota)? I'm sure some of us wouldn't mind running our own instances to take some of the traffic off the appspot service. :)
@msh100
Contributor

msh100 commented Sep 20, 2015

I have tried with my own oauth handler, and get the same error. I think there's still an issue with Amazon.
@vampywiz17

vampywiz17 commented Sep 20, 2015

Same error with me.. I talk the amazon support. The answer:

"Hello,

I'm sorry to hear about the trouble you've in using Cloud Drive Web and PC application.

At this point of time, we are currently facing the issue and our technical team is working on it to resolve the at the earliest.

Please try again over the next few hours. Errors like this are usually corrected shortly after they're reported.

Before you try, please signout from the application and then signin and try again.

One of the benefits we try very hard to offer our customers is convenience. I'm very sorry for the inconvenience you experienced.

Thanks for your patience and understanding in this matter. We look forward to see you soon again.

We'd appreciate your feedback. Please use the links below to tell us about your experience today."
@jon-rolfe

jon-rolfe commented Sep 20, 2015

@yadayada In 17 hours when the quota is reset, won't there be an even larger spike in requests from the 24+ hour downtime? If so... it'll hit the quota almost instantly again. Repeat ad infinitum.
@msh100
Contributor

msh100 commented Sep 20, 2015

Ignore me, I was able to sync when refreshing the oauth_data, maybe it had some error in it

@jon-rolfe has a very good point. This may be the time to provide real documentation on setting own oauth handler.
@jamesrascal

jamesrascal commented Sep 20, 2015

Can we setup a donate portal or something to bring this back online?
@msh100
Contributor

msh100 commented Sep 20, 2015

Donations won't help this. Check #74 (comment) for using your own oauth app - the steps with the webserver are unneseccary.
@conihorse

conihorse commented Sep 20, 2015

@msh100 - broken link in #74 to the oauth source - do you know anywhere else it's available?
@msh100
Contributor

msh100 commented Sep 20, 2015

Heh interesting, that should still be there - give me a moment

Actually never mind, that step is unnecessary - skip the step about a local webserver. You can just paste the URL you are redirected to into acd_cli
@vampywiz17

vampywiz17 commented Sep 20, 2015

I try to create own oauth (skip the webserver step) but i get a error if i try to run acd_cli sync:

The firefox cant connect the "locashost" server

What i do it wrong?
@msh100
Contributor

msh100 commented Sep 20, 2015

That's fine. Just copy the url Firefox is trying to make you go to and paste in to acd_cli

Sent from my iPhone

On 20 Sep 2015, at 17:37, vampywiz17 <notifications@github.commailto:notifications@github.com> wrote:

I try to create own oauth (drip the webserver step) but i get a error if i try to run acd_cli sync:

The firefox cant connect the "locashost" server

What i do it wrong?

Reply to this email directly or view it on GitHubhttps://github.com/yadayada/acd_cli/issues/147#issuecomment-141808827.
@vampywiz17

vampywiz17 commented Sep 20, 2015

ok but where? :)

and how...
@msh100
Contributor

msh100 commented Sep 20, 2015

When it launches your browser it should be waiting for you to input the URL you were redirected to, so just go back to the terminal
@vampywiz17

vampywiz17 commented Sep 20, 2015

Ohh i see now is working! Thanks a lot!

One question: next time, if i try to sync, (like restart computer) it is need to do this steps again?
@msh100
Contributor

msh100 commented Sep 20, 2015

No, if you check ~/.cache/acd_cli/, there's now a client_data file. This contains your application data. It will remain there forever 😄
@yadayada
Owner

yadayada commented Sep 20, 2015

If someone is desperate, I put up a backup on https://acd-cli-oa-backup.appspot.com

diff --git a/acdcli/api/oauth.py b/acdcli/api/oauth.py
index 2bc1e60..b8ffd99 100644
--- a/acdcli/api/oauth.py
+++ b/acdcli/api/oauth.py
@@ -124,7 +124,7 @@ class OAuthHandler(object):


 class AppspotOAuthHandler(OAuthHandler):
-    APPSPOT_URL = 'https://tensile-runway-92512.appspot.com/'
+    APPSPOT_URL = 'https://acd-cli-oa-backup.appspot.com'

     def __init__(self, path):
         super().__init__(path)
@vampywiz17

vampywiz17 commented Sep 20, 2015

Great!

Thanks!
@fiunchinho

fiunchinho commented Sep 20, 2015

Changing appspot url now I get

Press a key if you have saved the "oauth_data" file into "/root/.cache/acd_cli".

15-09-20 23:33:14.822 [CRITICAL] [acdcli.api.client] - Error getting endpoint data. Response: <html><body><b>Http/1.1 Service Unavailable</b></body> </html>
Traceback (most recent call last):
  File "./bin/acdcli", line 9, in <module>
    load_entry_point('acdcli==0.3.0a6', 'console_scripts', 'acdcli')()
  File "/root/probando/bin/acd_cli.py", line 1241, in main
    acd_client = client.ACDClient(CACHE_PATH)
  File "/root/probando/lib/python3.3/site-packages/acdcli/api/client.py", line 34, in __init__
    self._load_endpoints()
  File "/root/probando/lib/python3.3/site-packages/acdcli/api/client.py", line 44, in _load_endpoints
    self._endpoint_data = self._get_endpoints()
  File "/root/probando/lib/python3.3/site-packages/acdcli/api/client.py", line 58, in _get_endpoints
    raise Exception
Exception
@msh100
Contributor

msh100 commented Sep 20, 2015

Yeh appears to be another Amazon problem
@RobinHoutevelts

RobinHoutevelts commented Sep 20, 2015

I could fetch my oath_data right now.
Amazon is just being very buggy :(

Edit: tried again later. Getting errors again :/
@memoz
Contributor

memoz commented Sep 21, 2015

Amazon's very sorry about what happened:
a
@dellipse

dellipse commented Sep 21, 2015

After trying to use the oauth data from the backup server, I am now getting the following errors:

Syncing...
15-09-21 00:18:58.246 [CRITICAL] [acd_cli] - Sync failed.
RequestError: 1000, HTTPSConnectionPool(host='cdws.us-east-1.amazonaws.com', port=443): Read timed out. (read timeout=30)
Done.

and

Syncing...
15-09-21 00:19:17.373 [CRITICAL] [acd_cli] - Sync failed.
RequestError: 503, [acd_cli] no body received.
Done.

Any idea what is causing these issues? Are they related to the oauth change? Do I need to modify a file on my machines to use the backup oauth APPSPOT_URL?
@vampywiz17

vampywiz17 commented Sep 21, 2015

Same error with me... I think it is not a settings problem your machine. It is amazon problem... I can't login to the website, so it is a global fault... I think
@yadayada
Owner

yadayada commented Sep 21, 2015

screenshot-amazon cloud drive - google chrome
@dellipse

dellipse commented Sep 21, 2015

Amazon is still having issues with CloudDrive, but if you call them, they are at least offering some promotional credits as a token...not that it helps, but....
@memoz
Contributor

memoz commented Sep 21, 2015

Seems that Amazon have fixed it by now.
@ug945
Contributor

ug945 commented Sep 21, 2015

acd
@dellipse

dellipse commented Sep 21, 2015

No, it is sporadic and not effecting everyone according to the CSR I spoke with.
@memoz
Contributor

memoz commented Sep 21, 2015

Well, I happened to be in one of the tiny spots when it sporadically worked...then I'm out of luck. It's best if Amazon tell us what happened afterwards.
@memoz
Contributor

memoz commented Sep 21, 2015 

...Syncing...
15-09-21 17:09:42.058 [WARNING] [acdcli.api.metadata] - End of change request not reached.
Done.

When this happens, acd_cli returns 0 and the cache is not updated. Maybe this error should produce a non-zero exit code?
@amaxcz

amaxcz commented Sep 21, 2015

ACD broken totally. Its allows upload sometime. but not listing of files. so, downtime ~14h for now :)
@memoz
Contributor

memoz commented Sep 21, 2015

At least 23h now according to GAE's spike. Their uptime just dropped to 99.7% this year. It better not be, but let's see if ACD crosses the 99% line...that would be 88h downtime.

yadayada added the AWS apocalpyse label Sep 21, 2015

@dellipse

dellipse commented Sep 21, 2015

Here is the latest sample response (outside of the 503 errors)

acdcli -v sync
15-09-21 12:23:10.983 [INFO] [acd_cli] - Plugin leaf classes: TestPlugin
15-09-21 12:23:10.983 [INFO] [acd_cli] - TestPlugin attached.
15-09-21 12:23:10.983 [INFO] [acdcli.api.client] - Initializing ACD with path "/YourHomePath/.cache/acd_cli".
15-09-21 12:23:10.983 [INFO] [acdcli.api.oauth] - AppspotOAuthHandler initialized
15-09-21 12:23:10.984 [INFO] [acdcli.cache.db] - Initializing cache with path "/YourHomePath/.cache/acd_cli".
15-09-21 12:23:11.003 [INFO] [acdcli.cache.db] - Cache not considered uninitialized.
15-09-21 12:23:11.005 [INFO] [acdcli.cache.db] - DB schema version is 1.
Syncing...
15-09-21 12:23:11.014 [INFO] [acdcli.api.metadata] - Getting changes with checkpoint "None".
15-09-21 12:23:11.014 [INFO] [acdcli.api.backoff_req] - POST "https://cdws.us-east-1.amazonaws.com/drive/v1/changes"
15-09-21 12:23:11.017 [INFO] [requests.packages.urllib3.connectionpool] - Starting new HTTPS connection (1): cdws.us-east-1.amazonaws.com
15-09-21 12:23:12.495 [INFO] [acdcli.api.metadata] - -1 pages, 0 nodes, 0 purged nodes in changes.
15-09-21 12:23:12.495 [WARNING] [acdcli.api.metadata] - End of change request not reached.
Done.

Not sure if we want to open another issue for the fact that the lack of data results in an "everything worked" result code of 0.
@jonas5

jonas5 commented Sep 21, 2015 

acdcli.api.common.RequestError: RequestError: 1004, Error refreshing authentication token: <html>
  <head>
    <title>Internal Server Error</title>
    <style>
      body {
        padding: 20px;
        font-family: arial, sans-serif;
        font-size: 14px;
      }
      pre {
        background: #F2F2F2;
        padding: 10px;
      }
    </style>
  </head>
  <body>
    <h1>Internal Server Error</h1>
    <p>The server has either erred or is incapable of performing
    the requested operation.</p>
    <pre>Traceback (most recent call last):
  File &quot;/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py&quot;, line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File &quot;/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py&quot;, line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File &quot;/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py&quot;, line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File &quot;/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py&quot;, line 1102, in __call__
    return handler.dispatch()
  File &quot;/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py&quot;, line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File &quot;/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py&quot;, line 570, in dispatch
    return method(*args, **kwargs)
  File &quot;/base/data/home/apps/s~tensile-runway-92512/1.387290494974093533/main.py&quot;, line 105, in post
    resp = urllib2.urlopen(AMAZON_OA_TOKEN_URL, urllib.urlencode(params), timeout=15)
  File &quot;/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py&quot;, line 127, in urlopen
    return _opener.open(url, data, timeout)
  File &quot;/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py&quot;, line 410, in open
    response = meth(req, response)
  File &quot;/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py&quot;, line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File &quot;/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py&quot;, line 448, in error
    return self._call_chain(*args)
  File &quot;/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py&quot;, line 382, in _call_chain
    result = func(*args)
  File &quot;/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py&quot;, line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: Bad Request
</pre>
  </body>
</html>
@vampywiz17

vampywiz17 commented Sep 21, 2015

The web connect start to normalize (not perfect, but sometimes loading well) but the acd_cli not working...
@dellipse

dellipse commented Sep 21, 2015

@yadayada I suggest that once things calm down with the ACD API side of things, you reach out to the AWS folks to see if the can reset your daily limit so you can continue to operate normally as this was caused by an Amazon issue, not your code. They have done similar things for me in the past, but your mileage may vary....
@vampywiz17

vampywiz17 commented Sep 21, 2015

@dellipse I think, lot of as now use own OAuth server (me, yes) so now the limit not the biggest problem.
@dellipse

dellipse commented Sep 21, 2015

@vampywiz17 True, but for some people it is a pain to set up their own OAuth server, so they still rely on his. If AWS resets his quota, that fixes THAT piece of the issue for everyone.

Now to get ACD to start working with their APIs. It would also be good for them to tell us what happened. It would be awesome if they used this format from meh.com, but I doubt they will.

At least they could use this when things go wrong.
@metalgeekstl

metalgeekstl commented Sep 21, 2015

Are we all still waiting for Amazon to get their collective S&#* together?
I'm still getting "[WARNING] [acdcli/api.metadata] - End of change request not reached."
@msh100
Contributor

msh100 commented Sep 21, 2015

Can we stop all the unproductive spam on this issue? My inbox is becoming full of useless information for a situation none of us have control over. I propose we only add stuff to this issue when there's something in acd_cli's hands. We all know Amazon Cloud Drive has issues now.
@memoz
Contributor

memoz commented Sep 22, 2015

Finally a successful sync. Anyone else?
@vampywiz17

vampywiz17 commented Sep 22, 2015

Yes, working again.

yadayada closed this Sep 22, 2015

@yadayada yadayada added a commit that referenced this issue Oct 10, 2015

@yadayada yadayada change backoff, oauth behavior 
Backoff request will fail on failed auth_callback to delay next try (#147).

OAuth handler creation will raise if local handler's client_data file exists
and init fails, so as not to mix appspot oauth data.
Also, show error if existing oauth_file is invalid.
 6adcad9

yadayada referenced this issue Sep 1, 2016

Closed

HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://api.amazon.com/auth/o2/token #391

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment