GDAX enableRateLimit:true yields "Rate limit exceeded"

[albertl2]

I enabled the rate limit using this bit of code:

exchange = ccxt.gdax ({ 'enableRateLimit':True, # let ccxt manage the xfer rate })

And then called my async loop like this:

asyncio.get_event_loop().run_until_complete(main(exchange))

Which yielded the following... I don't understand asyncio very well, and I'm unsure how to enforce sequential, timed calls to the exchange. I thought enabling the rate limit on the exchange would take care of this?

DDoSProtection                            Traceback (most recent call last)
<ipython-input-32-11176782990b> in <module>()
      3 })
      4 
----> 5 asyncio.get_event_loop().run_until_complete(main(exchange))

/usr/lib/python3.6/asyncio/base_events.py in run_until_complete(self, future)
    465             raise RuntimeError('Event loop stopped before Future completed.')
    466 
--> 467         return future.result()
    468 
    469     def stop(self):

<ipython-input-31-a54051e9d25c> in main(exchange)
     33         ETHBTC = await getTicker(exchange,'ETH/BTC')
     34 
---> 35         LTCUSD = await getTicker(exchange,'LTC/USD')
     36         LTCBTC = await getTicker(exchange,'LTC/BTC')
     37 

<ipython-input-31-a54051e9d25c> in getTicker(exchange, ticker)
      6 
      7 async def getTicker(exchange,ticker):
----> 8     price = await exchange.fetchTicker(ticker)
      9     return price['last']
     10 

~/.local/lib/python3.6/site-packages/ccxt/async/gdax.py in fetch_ticker(self, symbol, params)
    179         }, params)
    180         ticker = await self.publicGetProductsIdTicker(request)
--> 181         quote = await self.publicGetProductsIdStats(request)
    182         timestamp = self.parse8601(ticker['time'])
    183         bid = None

~/.local/lib/python3.6/site-packages/ccxt/async/gdax.py in request(self, path, api, method, params, headers, body)
    467 
    468     async def request(self, path, api='public', method='GET', params={}, headers=None, body=None):
--> 469         response = await self.fetch2(path, api, method, params, headers, body)
    470         if 'message' in response:
    471             raise ExchangeError(self.id + ' ' + self.json(response))

~/.local/lib/python3.6/site-packages/ccxt/async/base/exchange.py in fetch2(self, path, api, method, params, headers, body)
     82         self.lastRestRequestTimestamp = self.milliseconds()
     83         request = self.sign(path, api, method, params, headers, body)
---> 84         return await self.fetch(request['url'], request['method'], request['headers'], request['body'])
     85 
     86     async def fetch(self, url, method='GET', headers=None, body=None):

~/.local/lib/python3.6/site-packages/ccxt/async/base/exchange.py in fetch(self, url, method, headers, body)
    104                 text = await response.text()
    105                 self.handle_errors(response.status, text, url, method, None, text)
--> 106                 self.handle_rest_errors(None, response.status, text, url, method)
    107         except socket.gaierror as e:
    108             self.raise_error(ExchangeError, url, method, e, None)

~/.local/lib/python3.6/site-packages/ccxt/base/exchange.py in handle_rest_errors(self, exception, http_status_code, response, url, method)
    342             error = AuthenticationError
    343         if error:
--> 344             self.raise_error(error, url, method, exception if exception else http_status_code, response)
    345 
    346     def handle_rest_response(self, response, url, method='GET', headers=None, body=None):

~/.local/lib/python3.6/site-packages/ccxt/base/exchange.py in raise_error(self, exception_type, url, method, error, details)
    239                 method,
    240                 url,
--> 241                 details,
    242             ]))
    243         else:

DDoSProtection: gdax GET https://api.gdax.com/products/LTC-USD/stats 429 {"message":"Rate limit exceeded"}

In [ ]:

[albertl2]

So, I just realized that using await ensures that the calls are sequential, but does nothing for the timing. Is that still handled by ccxt via the enableRateLimit:True setting? Or do I also need to handle the rate limit in my logic?

[kroitor]

Is that still handled by ccxt via the enableRateLimit:True setting?

Should be.

Let us know if this example does not work for you:

https://github.com/ccxt/ccxt/blob/master/examples/py/async-gdax-fetch-ticker-continuously.py

If it raises a DDoSProtection exception, then please report how long it takes to crash. Also, it would be much easier for us to help you, if you could share a snippet of your code to reproduce the problem. I'm closing this for now, feel free to reopen it if needed, or just ask questions if any. Thx!

[albertl2]

@kroitor I ran the code from the example and it produced the same results. The starting time-stamp (from the output of the program itself) was 2017-12-09T15:00:34.834Z. The final time-stamp was 2017-12-09T15:04:00.717Z.

Also, There were 100 calls to GDAX from 9:00 to 9:04, about 25 a minute... Their docs say they'll take up to 3 calls per second, so I don't know what is going on.

Are there any other tests that it would be helpful for me to run? Or other code that you need to see?

DDoSProtection                            Traceback (most recent call last)
<ipython-input-2-9fae7563c369> in <module>()
     29 })
     30 
---> 31 asyncio.get_event_loop().run_until_complete(main(exchange, 'LTC/USD'))

/usr/lib/python3.6/asyncio/base_events.py in run_until_complete(self, future)
    465             raise RuntimeError('Event loop stopped before Future completed.')
    466 
--> 467         return future.result()
    468 
    469     def stop(self):

<ipython-input-2-9fae7563c369> in main(exchange, symbol)
     16         print(exchange.iso8601(exchange.milliseconds()), 'fetchin', symbol, 'ticker from', exchange.name)
     17         # this can be any call instead of fetch_ticker, really
---> 18         ticker = await exchange.fetch_ticker(symbol)
     19         print(exchange.iso8601(exchange.milliseconds()), 'fetched', symbol, 'ticker from', exchange.name)
     20         print(ticker)

~/.local/lib/python3.6/site-packages/ccxt/async/gdax.py in fetch_ticker(self, symbol, params)
    179         }, params)
    180         ticker = await self.publicGetProductsIdTicker(request)
--> 181         quote = await self.publicGetProductsIdStats(request)
    182         timestamp = self.parse8601(ticker['time'])
    183         bid = None

~/.local/lib/python3.6/site-packages/ccxt/async/gdax.py in request(self, path, api, method, params, headers, body)
    467 
    468     async def request(self, path, api='public', method='GET', params={}, headers=None, body=None):
--> 469         response = await self.fetch2(path, api, method, params, headers, body)
    470         if 'message' in response:
    471             raise ExchangeError(self.id + ' ' + self.json(response))

~/.local/lib/python3.6/site-packages/ccxt/async/base/exchange.py in fetch2(self, path, api, method, params, headers, body)
     82         self.lastRestRequestTimestamp = self.milliseconds()
     83         request = self.sign(path, api, method, params, headers, body)
---> 84         return await self.fetch(request['url'], request['method'], request['headers'], request['body'])
     85 
     86     async def fetch(self, url, method='GET', headers=None, body=None):

~/.local/lib/python3.6/site-packages/ccxt/async/base/exchange.py in fetch(self, url, method, headers, body)
    104                 text = await response.text()
    105                 self.handle_errors(response.status, text, url, method, None, text)
--> 106                 self.handle_rest_errors(None, response.status, text, url, method)
    107         except socket.gaierror as e:
    108             self.raise_error(ExchangeError, url, method, e, None)

~/.local/lib/python3.6/site-packages/ccxt/base/exchange.py in handle_rest_errors(self, exception, http_status_code, response, url, method)
    342             error = AuthenticationError
    343         if error:
--> 344             self.raise_error(error, url, method, exception if exception else http_status_code, response)
    345 
    346     def handle_rest_response(self, response, url, method='GET', headers=None, body=None):

~/.local/lib/python3.6/site-packages/ccxt/base/exchange.py in raise_error(self, exception_type, url, method, error, details)
    239                 method,
    240                 url,
--> 241                 details,
    242             ]))
    243         else:

DDoSProtection: gdax GET https://api.gdax.com/products/LTC-USD/stats 429 {"message":"Rate limit exceeded"}

[kroitor]

Also, There were 100 calls to GDAX from 9:00 to 9:04, about 25 a minute... Their docs say they'll take up to 3 calls per second, so I don't know what is going on.

Yeah, that's odd, indeed. Seems like their actual rate limits differ from what is in their docs... Or it might be a different layer of DDoS protection, say, from Cloudflare...

Are there any other tests that it would be helpful for me to run? Or other code that you need to see?

Let me do the testing on my side, I'll try to reverse-engineer their rate limit, and I'll get back to you with my results shortly.

[paskals]

I've been using their API since the summer. Last week or so I started getting "rate limit exceeded" exceptions out of nowhere. I reduced my rate to around 3 request per second, but today even this rate results in exceptions. They have scheduled maintenance for tomorrow (hopefully to address this).

[kroitor]

@paskal91 thank you for the feedback on this! Makes much more sense now. Seemingly, all exchanges are being overloaded.

[npomfret]

I've started getting this rate limit error also. Maybe they reduce the limits without telling anyone?

[paskals]

If that's the case, they haven't updated their docs. The spec still says 5 requests per second. Currently, even once per second seems unstable...

[kroitor]

Thanks for your reports, guys!
Can you please try and compare these two scripts with each other:

• order book: https://github.com/ccxt/ccxt/blob/master/examples/py/async-gdax-fetch-order-book-continuously.py
• ticker: https://github.com/ccxt/ccxt/blob/master/examples/py/async-gdax-fetch-ticker-continuously.py

It's the same script, different endpoints. Do you get the same result with the orderbook as with the ticker? To me it looks like their rate limits are endpoint-specific, i'm getting "rate limit exceeded" with the ticker, no errors with the order book. Would be happy to get more feedback from you.