ApiCallError - OSError("(104, 'ECONNRESET')")

[ArielS1]

Hi,
I'm a script on an amazon EC2 with operating system Ubuntu Server 18.04 LTS (HVM).
Firebase SDK version: 2.14.0
Python version: 3.7
Firebase product: database

once in 12-48 hours while i try to use a noraml .get() command on different parts of my code i get the following exception:

ApiCallError
('Connection aborted.', OSError("(104, 'ECONNRESET')"))
event:   File "/home/script.py", line 29, in script
    database.reference('data').order_by_child('parameters/name').get()
  File "/usr/local/lib/python3.7/site-packages/firebase_admin/db.py", line 612, in get
    result = self._client.body('get', self._pathurl, params=self._querystr)
  File "/usr/local/lib/python3.7/site-packages/firebase_admin/_http_client.py", line 93, in body
    resp = self.request(method, url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/firebase_admin/db.py", line 884, in request
    raise ApiCallError(_Client.extract_error_message(error), error)

sometimes it happens with order_by_child get and sometimes on a normal get.

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[hiranya911]

A few low-level errors are to be expected when interacting with RTDB over a long period (due to backend server restarts, and other network issues). You can catch the error and retry when this happens. In the future, the SDK will also do some automatic retrying (see #247) when something lik this happens.

[hiranya911]

Now that #247 is merged, future versions of the SDK will automatically retry upon IO errors.

[ArielS1]

Thanks
I got the package with the relevant commit and since that i haven't got that exception anymore,
But now i encountered a new one:
503 Server Error: Service Unavailable for url: https://xxx.firebaseio.com/xxx.json?print=silent Reason: traceback: File "/home/ubuntu/xxx.py", line 101, in xx 'xx': yy File "/usr/local/lib/python3.7/site-packages/firebase_admin/db.py", line 339, in update self._client.request('patch', self._add_suffix(), json=value, params='print=silent') File "/usr/local/lib/python3.7/site-packages/firebase_admin/db.py", line 884, in request raise ApiCallError(_Client.extract_error_message(error), error)

I believe that this kind of exception should be also handled using retries, because a minute after the same update() worked

[hiranya911]

@kill4food we do retry for 503 errors now (it was implemented in the same PR referenced above). We retry up to 4 times with exponential backoff before giving up. That adds up to a retry window of around 8 seconds. In the above case the error probably lasted longer than that. Keep us posted on how often you see this sort of errors. If 503's are frequent, we can look into further extending the retry window.

[hiranya911]

Oh, actually it turns out requests does not retry some HTTP methods like PATCH by default. Probably because some of those methods are considered not safe to retry. I'll see if we can override that.

[hiranya911]

These are the methods that are retried by default: https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry.DEFAULT_METHOD_WHITELIST

[ArielS1]

About the frequency of this issue, i got it twice in a 3 months period
But if it could be better if patch exceptions could be retried, since update() is a very common method

[ArielS1]

Actually i got now the same exception after less than 24 hours from the last time, so maybe it's more frequent than i tought