Download error can cause stalling and error looping

[rezan]

Ran into a situation where a single download error caused an error loop and hung cvmfs2. The cause are these Backoff() calls in VerifyAndFinalize() which sleep between retries:

cvmfs/cvmfs/network/download.cc

Line 1602 in 6b4ccc1

Backoff(info);

This is all done in the libcurl event loop in MainDownload(). When this happens and a single download sleeps during its retry, all downloads stall out due to the event loop not being run. This then causes all kinds of new errors like timeouts and TooSlow errors. These then trigger more Backoffs() and you are potentially stuck in a forever fail/retry loop depending on the conditions.

Possibly related to #3432 and #3378.

[vvolkl]

Thanks for this report! With regards to the connection to the linked issues: Did the cvmfs2 process still respond to SIGKILL when you encountered this bug?

[rezan]

Hmm, good question, I didn't test that. However, I can confirm my user process moved into a <defunct> state and couldn't be killed. I think thats cause it was stuck in a read() syscall. I can reproduce this pretty easily, so I will test the cvmfs2 process later and report back.

[rezan]

I was able to kill the process. I was then able to re-mount and recover. So no issues there. Note that to trigger this I just have to generate a decent amount of CVMFS load and then kill a proxy server. Thats enough to get 1 thread into a Backoff() and the stall/retry/fail loop starts.

[HereThereBeDragons]

for replication:
have at least 1 private proxy, set the number of retries very high (CVMFS_MAX_RETRIES=1000)

1. start some downloads
2. kill the proxy --> you should see that it goes into backoff and replies with "too slowly"
3. reenable the proxy --> see that you are still in the backoff and never recover

Fix is in #3556 but only for the new introduced parallel decompression as this puts the backoff outside the sequential handling of curl callbacks

Edit:
It does not seem to be the same as #3432 and #3378 - but it should be investigated if #3556 might fix those problems anyway