Getting lots of timeout errors

[AlverGant]

First of all thank you for the python wrapper API.

I am running the following code on a raspberry pi and getting lots of timeout errors. Can you please help me?

Here is the code:

#!/usr/bin/env python3
import time
import pickle
from portfolio import portfolio
from coinmarketcapapi import CoinMarketCapAPI, CoinMarketCapAPIError

cmc = CoinMarketCapAPI('****deleted***')

# API requests limits imposed by Coinmarketcap on free API use
max_requests_per_minute = 30.0
max_requests_per_day = 300.0

# Timers to enforce free API usage
market_start_hour = 0.0
market_end_hour = 24.0
minutes_active = (market_end_hour - market_start_hour) * 60.0 # number of minutes in the market operation 8:00 to 24:00
forced_inactivity1 = 60.0 / max_requests_per_minute # do not exceed max_requests_per_minute
# do not exceeed max_requests_per_day
forced_inactivity2 = len(portfolio)*60.0*((minutes_active/max_requests_per_day)-1.0/max_requests_per_minute)
# BATCH OF API_REQUESTS + forced_inactivity2 + BATCH OF API_REQUESTS ...

def loop():
    total_now = 0
    total_yesterday = 0
    for i,amount in portfolio.items():
        ticker = cmc.cryptocurrency_quotes_latest(id=i,convert='BRL')
        total_now += float(ticker.data[i]['quote']['BRL']['price']) * float(amount)
        total_yesterday += float(ticker.data[i]['quote']['BRL']['price']) * float(amount) - (float(ticker.data[i]['quote']['BRL']['price']) * float(ticker.data[i]['quote']['BRL']['percent_change_24h']) / 100.0 * float(amount))
        time.sleep(forced_inactivity1)
    time.sleep(forced_inactivity2)
    gains = total_now - total_yesterday
    with open('config.ini', 'wb') as configfile:
        pickle.dump(gains,configfile)

def main():
    while True:
        loop()

if __name__=="__main__":
    main()

Here are the errors I am getting:
Jan 08 17:59:31 raspberrypi python3[11965]: Traceback (most recent call last):
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
Jan 08 17:59:31 raspberrypi python3[11965]: six.raise_from(e, None)
Jan 08 17:59:31 raspberrypi python3[11965]: File "", line 3, in raise_from
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
Jan 08 17:59:31 raspberrypi python3[11965]: httplib_response = conn.getresponse()
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
Jan 08 17:59:31 raspberrypi python3[11965]: response.begin()
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3.9/http/client.py", line 307, in begin
Jan 08 17:59:31 raspberrypi python3[11965]: version, status, reason = self._read_status()
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3.9/http/client.py", line 268, in _read_status
Jan 08 17:59:31 raspberrypi python3[11965]: line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3.9/socket.py", line 704, in readinto
Jan 08 17:59:31 raspberrypi python3[11965]: return self._sock.recv_into(b)
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3.9/ssl.py", line 1241, in recv_into
Jan 08 17:59:31 raspberrypi python3[11965]: return self.read(nbytes, buffer)
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3.9/ssl.py", line 1099, in read
Jan 08 17:59:31 raspberrypi python3[11965]: return self._sslobj.read(len, buffer)
Jan 08 17:59:31 raspberrypi python3[11965]: TimeoutError: [Errno 110] Connection timed out
Jan 08 17:59:31 raspberrypi python3[11965]: During handling of the above exception, another exception occurred:
Jan 08 17:59:31 raspberrypi python3[11965]: Traceback (most recent call last):
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
Jan 08 17:59:31 raspberrypi python3[11965]: resp = conn.urlopen(
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
Jan 08 17:59:31 raspberrypi python3[11965]: retries = retries.increment(
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
Jan 08 17:59:31 raspberrypi python3[11965]: raise six.reraise(type(error), error, _stacktrace)
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
Jan 08 17:59:31 raspberrypi python3[11965]: raise value
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
Jan 08 17:59:31 raspberrypi python3[11965]: httplib_response = self._make_request(
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 447, in _make_request
Jan 08 17:59:31 raspberrypi python3[11965]: self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 353, in _raise_timeout
Jan 08 17:59:31 raspberrypi python3[11965]: raise ReadTimeoutError(
Jan 08 17:59:31 raspberrypi python3[11965]: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='pro-api.coinmarketcap.com', port=443): Read timed out. (read timeout=None)
Jan 08 17:59:31 raspberrypi python3[11965]: During handling of the above exception, another exception occurred:
Jan 08 17:59:31 raspberrypi python3[11965]: Traceback (most recent call last):
Jan 08 17:59:31 raspberrypi python3[11965]: File "/home/pi/.local/lib/python3.9/site-packages/coinmarketcapapi/init.py", line 172, in __get
Jan 08 17:59:31 raspberrypi python3[11965]: response = self.__session.get(url, params=kwargs)
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 555, in get
Jan 08 17:59:31 raspberrypi python3[11965]: return self.request('GET', url, **kwargs)
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
Jan 08 17:59:31 raspberrypi python3[11965]: resp = self.send(prep, **send_kwargs)
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
Jan 08 17:59:31 raspberrypi python3[11965]: r = adapter.send(request, **kwargs)
Jan 08 17:59:31 raspberrypi python3[11965]: File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
Jan 08 17:59:31 raspberrypi python3[11965]: raise ReadTimeout(e, request=request)
Jan 08 17:59:31 raspberrypi python3[11965]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='pro-api.coinmarketcap.com', port=443): Read timed out. (read timeout=None)
Jan 08 17:59:31 raspberrypi python3[11965]: During handling of the above exception, another exception occurred:
Jan 08 17:59:31 raspberrypi python3[11965]: Traceback (most recent call last):
Jan 08 17:59:31 raspberrypi python3[11965]: File "/home/pi/cripto/getcripto.py", line 40, in
Jan 08 17:59:31 raspberrypi python3[11965]: main()
Jan 08 17:59:31 raspberrypi python3[11965]: File "/home/pi/cripto/getcripto.py", line 37, in main
Jan 08 17:59:31 raspberrypi python3[11965]: loop()
Jan 08 17:59:31 raspberrypi python3[11965]: File "/home/pi/cripto/getcripto.py", line 26, in loop
Jan 08 17:59:31 raspberrypi python3[11965]: ticker = cmc.cryptocurrency_quotes_latest(id=i,convert='BRL')
Jan 08 17:59:31 raspberrypi python3[11965]: File "/home/pi/.local/lib/python3.9/site-packages/coinmarketcapapi/init.py", line 239, in cryptocurrency_quotes_latest
Jan 08 17:59:31 raspberrypi python3[11965]: return self.__get(
Jan 08 17:59:31 raspberrypi python3[11965]: File "/home/pi/.local/lib/python3.9/site-packages/coinmarketcapapi/init.py", line 190, in __get
Jan 08 17:59:31 raspberrypi python3[11965]: self.__logger.warning(e)
Jan 08 17:59:31 raspberrypi python3[11965]: AttributeError: 'NoneType' object has no attribute 'warning'

I am not sure if its a network error but wifi connections seems stable on the raspberry pi looking at the logs.

Best regards

[AlverGant]

Timeouts are ocurring approximately every 45 minutes

[rsz44]

Hello !

First of all, thank you for the care you took with the Issue and sorry for the delay in responding.

You point out an error in the library which I will try to fix as soon as possible !

Indeed, line 190 of the library, self.__logger.warning(e) does not check if __logger is set to None. Whereas line 132 self.__logger = kwargs.get('logger', None) sets the logger to None if it is not specified.

To clean this error on your side, I invite you to pass a logger to the CoinMarketCapAPI constructor. In the code you gave me, you can do it like this (line 7 of your code) :

# [...]
import logging

cmc = CoinMarketCapAPI('****deleted***', logger=logging)
# [...]

However, this will not solve your problem directly as it seems that the server fails to return a response.

It seems at first sight that it is indeed a network problem. Indeed, in the library the errors of the API and the intermediate libraries are [supposed to] be caught and overridden.

We can see that the problem comes from: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='pro-api.coinmarketcap.com', port=443): Read timed out. (read timeout=None)

The server does not seem to be returning a response to your request at this time.

It seems that your program is asking the api for information at an "optimized" and relatively high rate (which coinmarketcap is rightly trying to make profitable).

What I would do is keep the result of your previous call in a variable in your loop and specifically catch this error. When it happens, I would save the previous call in a log file to check its status and remaining credits.

I would do something like this (from line 22 of your code, with the previous modification):

# [...] your previous code
def loop():
    total_now = 0
    total_yesterday = 0

    previous_ticker = None # added line

    for i,amount in portfolio.items():

        try:  # added line
            ticker = cmc.cryptocurrency_quotes_latest(id=i,convert='BRL')
        except CoinMarketCapAPIError as e:  # added line
            r = e.rep  # added line
            logging.error(f"Current ticker error : {repr(r.error)}")  # added line
            logging.error(f"Current ticker status : {repr(r.status)}")  # added line
            logging.error(f"Current ticker data : {repr(r.data)}")  # added line
            if previous_ticker:
                logging.error(f"Previous ticker credit_count : {repr(previous_ticker.credit_count)}")  # added line
                logging.error(f"Previous ticker status : {repr(previous_ticker.status)}")  # added line

        # [...] the 'sleep' code between

        previous_ticker = ticker # added line
# [...] your code that follows

According to the official Coinmarketcap API documentation, the server is supposed to return an HTTP error code 429, for example 1008 for, I quote: "1008 [API_KEY_PLAN_MINUTE_RATE_LIMIT_REACHED]". But as I have seen before, the documentation may be slightly out of sync with reality. It is possible that the server simply does not respond to requests beyond a certain threshold.

If after that, everything seems correct in terms of credits consumed, I would try to connect the raspberry to the ethernet, as close as possible to your box (avoiding any intermediate network device) just to check that this does not come from the network configuration. The 45 minute delay seems strange and it seems to be a "noise" from your network, especially if everything is ok on the server side.

I'll let you tell me how it goes and if you were able to make progress on your side. I look forward to your reply. I hope to be able to help you with your concerns.

I wish you a good day and happy coding !

[AlverGant]

Hi!

Thank you for your detailed reply!
I ran the modified code on my laptop, still connected to wifi but with a pretty stable connection and the errors are now gone!
Testing with the raspberry pi and it appears that problem is solved too!!
Don't know why I was gettings timeouts at 45 minutes intervals
I am trying to enforce CoinMarketCap API free rates with the sleep timers.

Before adding the logging I was getting timeout erros even with the laptop

Thank you very much!

[AlverGant]

Oops, I am still getting the same errors with no logging messages from the API in the try catch block.
Will try to connect the Pi through ethernet

[rsz44]

It will be some time before I update the API that allows me to fix the logger. I still hope that you have been able to solve the problem.

There were two invalid conditions regarding the logger which could be None and the class still called the logger. This is normally fixed in version 0.4 which I have just distributed. In this new version, the error should be more readable now.

If in doubt, I would advise always encapsulating via try...catch a piece of code calling for an external request and especially if this software is intended to run continuously. This seems to me to be THE simple practice to run-time problems.

However, this will not solve the timeout, which seems very local to me when I read it again today. Since then, no one else has come forward or commented on the same problem and this also reinforces this point to me. In your log, we see that a request is not successful and that a timeout is raised. This may be due to your network, but also to your raspberry (if there is an overload) or perhaps the server that skipped your requests for some reason. I can't give you a clear answer on this nor give you a software answer here other than to raise the error correctly, this package being specific to the CoinMarketCap API.

I have made a change about the first bug and I tried my best to answer you in my previous post. I am closing this ticket for now and I leave it to you or anyone else involved to continue, to respond or to reopen a new one if you deem it necessary.

Wish you a good continuation,