GitLab API call resulting in RemoteDisconnected Exception

[rossbeehler]

Have a really large GitLab.com (SaaS) project structure (3500+ projects), and when running gitlabform over the entire structure, we get a small handful of these errors:

Warning: Error occurred while processing project [project_url], exception:
('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Aside from adding some sort of retry mechanism in gitlabform/gitlab/core.py, any ideas of what might help us prevent this issue?

[gdubicki]

Hey, @rossbeehler!

We already have retries. Aren’t they covering this kind of error? Perhaps if you increase app verbosity level you will see that.

[gdubicki]

I have dug deeper into this.

The retry mechanism is not applied in all cases by urllib3.

One case are the POST requests which will not be retried as they are not idempotent.

To be precise, if the request was a POST but it was not yet sent, f.e. a connection timeout occurred while making the connection (urllib3.exceptions.ConnectTimeoutError), then the request will be retried.

But in the case of your RemoteDisconnected('Remote end closed connection without response') error, the request has already been sent to the server, so it might have actually processed it and the error might have only happened while replying.

The only reasonable way to handle such error then would be retrying processing of the whole section for a given entity. I have tried to do that in #497.

[gdubicki]

I merged the PR mentioned above and released v3.4.2 with the improvement. Let me know if you still observe errors with this version, @rossbeehler!