[ERROR] Uploading/overwriting Code: 429 "Rate exceeded"

[wdenx]

Hello,

recently I see a large number of "Rate exceeded" error messages, like:

 [ERROR] [acd_cli] - Uploading "XXX" failed. Code: 429, msg: {"logref":"xxx","message":"Rate exceeded","code":""}

or

[ERROR] [acd_cli] - Error overwriting file. Code: 429, msg: {"logref":"xxx","message":"Rate exceeded","code":""}

Is there anything I can do to avoid / work around this? What exactly does this error mean?

[yadayada]

I've seen this irregularly happen a few times, but I'm not completely sure why the error occurs. It may be that the client has exceeded the request limit for a given time interval or the waiting time before starting a consecutive request was too low.

Anyway, a quick workaround is to use the --max-retries argument for both uploading/overwriting.

[wdenx]

I forgot to mention that I was running the upload with "-r 3", but this did not help. This only happened when I tried uploading a whole directory hierarchy; when I later re-run the upload for the individual files that failed in this attempt, none of them showed any problems.

[yadayada]

In that case, I cannot reproduce the exact pattern of the errors that occurred.

You might want to try adding a forced waiting interval to the BackOffRequest class in acdcli/api/common.py like so (I added the last two lines):

    @classmethod
    def _wait(cls):
        with cls.__lock:
            duration = cls.__next_req - time.time()
        if duration > 5:
            logger.warning('Waiting %fs because of error(s).' % duration)
        logger.debug('Retry %i, waiting %f secs' % (cls.__retries, duration))
        if duration > 0:
            sleep(duration)
        else:
            sleep(random.random())

But it's a bit of a shot in the dark...

[msh100]

I have started hitting this on almost every request. Even acd_cli sync whilst using a proxy.

It seems like a genuine API rate limit to me, do you have any more insight?

[yadayada]

I think there are limitations on non-reviewed applications. I submitted my "app", which is running on https://tensile-runway-92512.appspot.com, in late May. It still hasn't been reviewed. I guess that might be it; although I haven't seen a large usage increase on the server.

There is still the option to get one's own security profile and "app" data, see
https://developer.amazon.com/public/apis/experience/cloud-drive/content/getting-started

This client data can then be put into a JSON file and used instead of the oauth server like described in the following link
https://github.com/yadayada/acd_cli#advanced-users

If that fixes your errors, than it's really a rate limitation problem. In any case, I'll ask the Amazon support.

[Linwood-F]

Do you know how the limits work, are they aggregated over all users of your app (in which case this will get worse) or specific to a given user, so that if we are using it lightly enough to not get errors it should keep running?

The link reads as though it is to become a commercial app (i.e. getting our own), or are you saying in that mode one (for ones's self) does not have these limits?

[msh100]

I think there are limitations on non-reviewed applications.

@Linwood-F : I think @yadayada simply doesn't know at the moment; which is understandable as there's zero documentation for this.

I will setup my own profile shortly and switch over when I next encounter the issue then update you with my results.

Edit: Also worth noting I am not the only person that encountered the issue at the same time.

[sneaker8650]

i 'm encountering the same problem now. Could you provide some help regarding your security profile app solution ? (if this is a solution at all) I'm not sure, which informations are needed for the "Security Profile Management" and how to continue.

[samcm]

Just started hitting this frequently over the last 24 hours after a couple weeks of smooth sailing. Supporting what @sneaker8650 has posted - if you could give us some further direction @yadayada that'd be awesome. Cheers.

[msh100]

I'd be very surprised if issues are not related ergo believe @yadayada's suggestion of it being an API limit with his app makes sense.

I am currently working on setting this up and documenting the process. I will post what I have shortly.

@sneaker8650 @samcm

[samcm]

That'd be great @msh100! Thanks mate.

[uwe-schwarz]

I'm actually running in the same problem now. Even after changing from the default client-data to my own there is no difference. I guess this is account (acd-account) related.

There is a thread in the amazon forums: https://forums.developer.amazon.com/forums/thread.jspa?threadID=5800&tstart=0

[msh100]

Note: If your Amazon account is not registered as a developer, you will first need to do that. https://developer.amazon.com/registration/profile.html.

Creating a New Security Profile

Start by going to create a new security profile.
Since you will only be using this yourself, it's not particularly important what you place in the fields.

Once you have submitted the form, you would have been given a client ID and a client secret.
You will need these for the next steps.

After this, you need to go to the "Web Settings" tab of your security profile and define "Allowed Origins" and "Allowed Return URLs".
These should both be set to http://localhost with no trailing slash.

Whitelisting your Application

In order to read/write data, you will need to request to whitelist your application.
Fill out the form, and the whitelist should be enabled almost instantly (check your emails).

Setup acd_cli to use new profile

On the computer you are using acd_cli, create a new file called ~/.cache/acd_cli/client_data.
This file should look like the following:

{
    "CLIENT_ID": "",
    "CLIENT_SECRET": ""
}

Configure Local Webserver

You will require a local app to handle the OAuth data.

Firstly, ensure you are not running any services on port 80 (you can start them back up after this process).
Alternatively alter the application to not listen on port 80 and configure your web server to proxy localhost requests (refer to your web server documentation).

Fetch the source of the web server used to handle requests.
This is a slightly altered version which listens on a web server, removes memcached, and fixes an API scope bug.

You'll need to ensure the Python 2 modules which are imported at the top of this file are available on the host (webapp2, urllib, urllib2, json, paste for the HTTP server).

Within this file, CLIENT_ID and CLIENT_SECRET are defined.
Replace those with your own values from the stages before.

CLIENT_ID = 'CLIENT ID HERE'
CLIENT_SECRET = 'CLIENT SECRET HERE'

To start the webserver, run python oauth_web.

Obtain your OAuth client data

That should be all, once you have the web server running and the client_data defined, you can run acd_cli sync.
A browser window should open asking if you want to authorise your application with Amazon.

Once this is done, you will have your oauth_data file.
This should be saved at ~/.cache/acd_cli/oauth_data.

You can now kill the web server process.

[yadayada]

@msh100 A local web server shouldn't be necessary. Just the oauth_data file needs to be deleted when switching the oauth handler.

[msh100]

@yadayada the local webserver has to return the oauth_data file surely?

[Linwood-F]

So is the inference that the limits hitting multiple people just now are some sort of in-aggregate limit for each "app", and by individually registering it as your app, we would each get such a limit?

Or is the white list aspect for running it yourself implying the rate limits are removed?

I spent a while trying to read the documentation on this, and while the process is laid out well, the actual limits themselves and when and how applied remained mysterious in what I read. Plus it's rather a concern that he submitted them 2 months ago and no review yet.

[msh100]

I think this has definitely helped my situation. I ran a test, the test did acd_cli sync 100 times with a 5 second sleep on two sets of config.

The shared application (shared.log) and my own (own.log).

$ grep -i 'rate exceeded' shared.log | wc -l
89
$ grep -i 'rate exceeded' own.log | wc -l
23

It's not perfect now but the shared application is consistently failing. I am pretty confident the rate limit is per application on Amazon.

I suspect the fact I still get failures is due to the fact I am the only user and would expect Amazon increase the limit with the active users of the app.

[yadayada]

@msh100 The local oauth handler will generate the oauth_data file from the URL that's supposed to be pasted on the initial run. I initially intended users to just copy the return URL from the error page that they will be redirected to after granting permisison to the app, e.g. http://localhost/?code=XXXXXXXXXXXXXXX&scope=clouddrive%3Aread+clouddrive%3Awrite

So, taking the info of #74 (comment) and #74 (comment), I assume this could be a cap to the increase of used storage of an account over a certain amount of time and another cap based on the app that's used to access the account. Meh, I don't like multivariate problems. :/

[msh100]

The local oauth handler will generate the oauth_data file from the URL that's supposed to be pasted on the initial run.

Not really sure either is ideal 😄

I assume this could be a cap to the increase of used storage of an account over a certain amount of time

I don't think they'd give errors for the API call to sync in that case.

[sneaker8650]

thanks for the tutorial. But it seems, switching to localoauthhandler did not work out in my case. Connection still gets dropped. :(

15-07-08 16:28:20.109 [INFO] [acd_cli] - Plugin leaf classes: StreamPlugin, TestPlugin
15-07-08 16:28:20.109 [INFO] [acd_cli] - StreamPlugin attached.
15-07-08 16:28:20.110 [INFO] [acd_cli] - TestPlugin attached.
15-07-08 16:28:20.110 [INFO] [acdcli.api.common] - Initializing acd with path "/root/.cache/acd_cli".
15-07-08 16:28:20.110 [INFO] [acdcli.api.oauth] - LocalOAuthHandler initialized.
15-07-08 16:28:20.125 [INFO] [requests.packages.urllib3.connectionpool] - Starting new HTTPS connection (1): drive.amazonaws.com
15-07-08 16:28:20.747 [CRITICAL] [acdcli.api.common] - Error getting endpoint data. Response: {"message":"Rate exceeded"}
Traceback (most recent call last):

[msh100]

That's a shame, I am out of suggestions. Using own application keys certainly improved the situation here but it's still flaky.

I will try and email Amazon also.

[uwe-schwarz]

From Sean Callahan over in the amazon forum:

Thank you for your post. At this time we do not publish rate limits for development and production. We have received multiple reports of developers receiving a 429 Rate Exceeded response from the Amazon Cloud Drive API. We are investigating the issue at the moment to identify the root cause. If you would like to email clouddrive-api-support@amazon.com and provide us with your client id that would be helpful. We will post more details when we have them.

Maybe if some post there client ids it might speed up the process.

[msh100]

We found the root cause of the issue. We enabled new throttling behavior and rolled back our changes as soon as we understood the impact to developers like yourself. The Amazon Cloud Drive API is now normalizing as the roll back nears 100% deployment. We apologize for this issue and inconvenience it may have caused you and your customers. Please feel free to email clouddrive-api-support@amazon.com if you have any further questions.

We still have to wonder if using your own App keys will be more reliable for individuals though.

[dcplaya]

I can confirm yesterday I had this issue and today it works and I am still using @yadayada way of auth

[LtRipley36706]

I can also confirm that I was having this issue and since Amazon's fix have not seen Rate exceeded errors since. I have opened a new issue #85 being unable to complete initial sync suspected to be due to large number of files/directories.

[yadayada]

I assume this is resolved permanently.

[jbuchbinder]

In case, for some reason, the source for the oauth app is inaccessible elsewhere, this is a backup copy:

#!/usr/bin/env python

import webapp2
import urllib
import urllib2
from google.appengine.api import memcache
import json

PRIVACY_HTML = ('<!DOCTYPE html><html><head><title>acd_cli_oa privacy info</title></head>'
                'This app does not save any personal data. You may view <a href="./src">the source code</a> to check. '
                'However, Google\'s App Engine will retain a log of your IP address, request URL and browser string.'
                '</html>')

AMAZON_OA_LOGIN_URL = 'https://amazon.com/ap/oa'
AMAZON_OA_TOKEN_URL = 'https://api.amazon.com/auth/o2/token'
LOCAL_REDIRECT_URI = 'http://localhost'

CLIENT_ID = memcache.get('CLIENT_ID')
CLIENT_SECRET = memcache.get('CLIENT_SECRET')

if not CLIENT_ID or not CLIENT_SECRET:
    with open("client.json") as cd:
        cl_d = json.load(cd)

    CLIENT_ID = cl_d['CLIENT_ID']
    CLIENT_SECRET = cl_d['CLIENT_SECRET']
    memcache.set('CLIENT_ID', CLIENT_ID)
    memcache.set('CLIENT_SECRET', CLIENT_SECRET)

REFRESH_TOKEN_KEY = 'refresh_token'
REDIRECT_URI_KEY = 'redirect_uri'

OAUTH_ST1 = {'client_id': CLIENT_ID,
             'response_type': 'code',
             # the 'read' scope is outdated and should be replaced by
             # 'read_all' for newly created security profiles
             'scope': 'clouddrive:read clouddrive:write',
             REDIRECT_URI_KEY: None}

OAUTH_ST2 = {'grant_type': 'authorization_code',
             'code': None,
             'client_id': CLIENT_ID,
             'client_secret': CLIENT_SECRET,
             REDIRECT_URI_KEY: None}

OAUTH_REF = {'grant_type': REFRESH_TOKEN_KEY,
             'refresh_token': None,
             'client_id': CLIENT_ID,
             'client_secret': CLIENT_SECRET,
             REDIRECT_URI_KEY: None}


def pp(string):
    return json.dumps(json.loads(string), indent=4, sort_keys=True)

def ppo(obj):
    return json.dumps(obj, indent=4, sort_keys=True)


class OauthHandler(webapp2.RequestHandler):
    def get(self):
        try:
            error = self.request.GET['error']
            error_desc = self.request.GET['error_description']

            self.response.headers['Content-Type'] = 'text/plain'
            self.response.write('Error: %s\nDescription: %s\n' % (error, error_desc))
            return
        except KeyError:
            pass

        try:
            code = self.request.GET['code']
            scope = self.request.GET['scope']
        except KeyError:
            params = OAUTH_ST1
            params[REDIRECT_URI_KEY] = self.request.host_url
            oauth_step1_url = AMAZON_OA_LOGIN_URL + '?' + urllib.urlencode(OAUTH_ST1)
            return webapp2.redirect(oauth_step1_url)

        # user has returned after oauth
        if code and scope:
            params = OAUTH_ST2
            params['code'] = code
            params[REDIRECT_URI_KEY] = self.request.host_url

            resp = urllib.urlopen(AMAZON_OA_TOKEN_URL, urllib.urlencode(params))

            self.response.headers.add('Content-Disposition', 'attachment; filename="oauth_data"')
            self.response.write(pp(resp.read()))
            return

    def post(self):
        try:
            ref = self.request.POST[REFRESH_TOKEN_KEY]
        except KeyError:
            self.response.set_status(400)
            err_resp = {'error': 'refresh token not supplied', 'request': self.request.__str__()}
            self.response.write(json.dumps(err_resp, sort_keys=True))
            return

        params = OAUTH_REF
        params[REFRESH_TOKEN_KEY] = ref
        params[REDIRECT_URI_KEY] = self.request.host_url
        resp = urllib2.urlopen(AMAZON_OA_TOKEN_URL, urllib.urlencode(params), timeout=15)

        code = resp.getcode()
        self.response.set_status(code)
        if code == 200:
            self.response.write(pp(resp.read()))
        else:
            err_resp = {}
            rt = resp.read()
            try:
                err_resp.update(json.loads(rt))
            except:
                err_resp = {'error': 'unknown error', 'response': str(rt)}
            err_resp['request'] = self.request.__str__()
            self.response.write(ppo(err_resp))


class PrivacyHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write(PRIVACY_HTML)


class SourceHandler(webapp2.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        with open(__file__[:-1]) as f:
            self.response.write(f.read())


app = webapp2.WSGIApplication([
    ('/', OauthHandler),
    ('/privacy', PrivacyHandler),
    ('/src', SourceHandler)
], debug=True)

[drgroot]

I'm noticing this issue again. I have tried to register my security profile (but not respsonse from amazon)
I also cannot figure out how to install the 'google.appengine.api' module.

any tips?

[kelek-]

I just got the same issue myself, while trying to use acd_cli mount.
This is what I get:
17-04-19 18:08:02.152 [CRITICAL] [acdcli.api.client] - Error getting endpoint data. Response: {"message":"Rate exceeded"}

[lloydie2009]

I'm also getting this today, I've tried removing all data in ". cache/acd_cli" and still get the same.

This is the same for US and UK accounts.

[nagye]

I am getting same error now. Saturday afternoon, 13/05. I cannot synch at all now!

[lloydie2009]

Interesting t know, you think Amazon are having issues with the API? Happens for both my US and UK acd. -------------------------------------------- On Sun, 14/5/17, nagye <notifications@github.com> wrote: Subject: Re: [yadayada/acd_cli] [ERROR] Uploading/overwriting Code: 429 "Rate exceeded" (#74) To: "yadayada/acd_cli" <acd_cli@noreply.github.com> Cc: "lloydie2009" <lloydie2009@googlemail.com>, "Comment" <comment@noreply.github.com> Date: Sunday, 14 May, 2017, 0:06 I am getting same error now. Saturday afternoon, 13/05. I cannot synch at all now! — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[MaxHayman]

Same here

[elPerstin]

and here, upgraded like 2 min ago to make sure its not an ubuntu- or acdcli-version-thing.
eu-account.

[arjunkc]

Same here

17-05-14 01:43:31.633 [CRITICAL] [acdcli.api.client] - Error getting endpoint data. Response: {"message":"Rate exceeded"}
Traceback (most recent call last):
  File "/usr/bin/acd_cli", line 11, in <module>
    load_entry_point('acdcli==0.3.2', 'console_scripts', 'acd_cli')()
  File "/usr/bin/acd_cli.py", line 1646, in main
    acd_client = client.ACDClient(CACHE_PATH, SETTINGS_PATH)
  File "/usr/lib/python3.6/site-packages/acdcli/api/client.py", line 46, in __init__
    self._load_endpoints()
  File "/usr/lib/python3.6/site-packages/acdcli/api/client.py", line 69, in _load_endpoints
    self._endpoint_data = self._get_endpoints()
  File "/usr/lib/python3.6/site-packages/acdcli/api/client.py", line 80, in _get_endpoints
    raise Exception
Exception

[Giantdouche33]

same here. Rclone works tho

[nagye]

Seems like it is something at Amazon´s end. When I try to create new oauth file, and click on the link, i get an Amazon error couldnt process request but they are working on it.