Download failure rate >50% (ESOCKETTIMEDOUT & Internal Server Error)

[littleskunk]

Package Versions

Replace the values below using the output from npm list storj-bridge.

Storjcli: 4.0.0 | Core: 6.0.10

Replace the values below using the output from node --version.

v6.9.2

Expected Behavior

Please describe the program's expected behavior. Include an example of your
usage code in the back ticks below if applicable.

Low failure rate. Failure reproduceable because farmer offline or any other common problem.

Actual Behavior

Please describe the program's actual behavior. Please include any stack traces
or log output in the back ticks below.

I download a few hundert files in a loop. I count Success and Failure for each file.
My best result was 50% failures. At the moment I get 66% failures.

Get Pointer and Create Token

[Mon Dec 19 2016 09:37:34 GMT+0000 (UTC)] [warn]   Request failed, reason: Internal Server Error

Only get Pointer

[Sat Dec 17 2016 23:52:22 GMT+0000 (UTC)] [warn]   Request failed, reason: Internal Server Error

My script will stop downloading files that failed 50 times in a row. -> Farmer must be online or I would have stoped downloading the file.

Last time I had one file that was on my own farmer but I can't find it anymore. As far as I remember the my farmer opens a data channel in time but the bridge sends me a timeout. Looks like a problem on the brige. If needed I can reproduce this test. My script already stoped a few files with ~200 download success but 50 failures in a row. I have the farmer nodeID and should be able to get his complete logfile.

[MeijeSibbel]

I am having the same issue, however, the success rate largely depends on the size of the files. File above 1-2GB have a download fail rate closer to 80-90%.

[littleskunk]

For my test I used my old single shard files without any mirrors.

[CJClaude]

Also seeing the warning. Of more concern is the termination error:

events.js:160
throw er; // Unhandled 'error' event
^

Error: write EPIPE
at exports._errnoException (util.js:1026:11)
at WriteWrap.afterWrite (net.js:799:14)

[braydonf]

After the last two releases, this should be largely resolved.

Improvements to download success rates will be an ongoing goal. There is still an outstanding issue with replication when nodes go offline to maintain enough mirrors for increasing download success that will be addressed.

re: EPIPE, this and some of the ECONNRESET needs more info to reproduce, and likely not related with bridge.

[MeijeSibbel]

@braydonf , This is not "largely resolved", i don't know if you guys tried to upload or download anything or not for the past 24-48 hours, but i haven't managed to download or upload a single shard due to the Internal server error, it has never been this bad, ever. A few timedouts is acceptable, but 100% failure for both downloading and uploading for over 48 hours is just not acceptable. If the internal server error is due to the bridge preventing us from uploading or downloading anything we should at least be notified about that in a normal manner in core-CLI.

[161chihuahuas]

@MeijeSibbel, you have almost certainly been rate limited and that is why. Of course we have tested.

[littleskunk]

I get the limit error message a few times but than I get only Internal Server error. I would expect a different Response from the bridge. -> #316