Json decode error during upload stage #2
Comments
|
It seems there are two problems here:
There is also a mention of HTTP/2 in the source code, but flathub seems to be using HTTP/1.1. |
|
So, part of the issue here is that we have an nginx front in front of the flat-manager service. Take the eclipse error above: This expects an error page to contain a json, but the actual error is from the nginx front: Clearly we should handle this better in the client, but in the end that doesn't really help us here. Early on in the new infra we got a lot of these for large delta uploads when the proxy timed out, but since then i've bumped the timeout and we see less of these. Its unclear exactly what caused this one though. "broken pipe", wth?? |
2019/03/01 15:11:01 [error] 12444#0: *14978582 sendfile() failed (32: Broken pipe) while sending request to upstream, client: 147.75.106.214, server: hub.flathub.org, request: "POST /api/v1/build/506/upload HTTP/1.1", upstream: "http://127.0.0.1:8080/api/v1/build/506/upload", host: "hub.flathub.org" This is not in the error logs, but the access.log for it ends with: 408 is Request Timeout, which means shutting down an unused connection, which apparently the client takes for a real error and aborts (purges). I think we should make the client handle this and re-try? This is weird though, this is the full log: I.e. at one point it returns 408, but that client continues anyway. But then suddently at the end it decides to purge? I'm not sure why this happened at all. |
|
So, for the broken pipe requests, i looked at the logs for the same time for the flat-manager service, and everything looks ok. Take the nginx error above for instance: In the service logs all related reqests to build 369 around this time are: So, there is nothing at 16:19:25, which seems to indicate that the request didn't even reach the service. Anyone has any idea why nginx would get a broken pipe error like this? |
|
Some references here: Nothing clear, but some mentions that doing the backend connection via a unix domain socket might help. We should maybe try that. |
|
the interesting thing it that it doesn't seem to happen on the freedesktop sdk flat-manager instance. Flathub:location /api {
# Allow uploading large builds, deltas, etc
client_max_body_size 0;
proxy_pass http://127.0.0.1:8080;
proxy_read_timeout 6000s;
add_header Vary Accept-Encoding;
}freedesktoplocation /api {
# Allow uploading large builds, deltas, etc
client_max_body_size 0;
proxy_pass http://127.0.0.1:8080;
proxy_read_timeout 600s;
add_header Vary Accept-Encoding;
}@alexlarsson have you tried to turn proxy_request_buffering off? |
|
I have not, i'm currently doing a major rework of the publish/update-repo part of flat-manager, so I'm focused on getting that done first. However, disabling that seems useful to try. Also, yes, flathub is quite overloaded since its serving a lot of request for the main repo while also generating deltas and handling new uploads. So, load it likely part of the issue. |
|
I can’t reproduce this anymore: https://flathub.org/builds/#/builders/32/builds/975. |
|
It's dependent on the load, it won't happen every time. |
|
Client now retries uploading if error returned an error. It's not technically a fix, but as it doesn't happen for FreeDesktop SDK and GNOME Nightly repo, it's hard to say if it's actual bug in flat-manager or Flathub hardware is not handling the load. So while it's not technically "fixed", I'll close it for now as it shouldn't be happening anymore, at least in a way preventing builds from being uploaded. |
doc22940
pushed a commit
to doc22940/free-desktop-sdk
that referenced
this issue
Nov 18, 2019
They are likely to fail when uploading to file flatpak/flat-manager#2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Since switching to flat-manager we have had some builds failed with:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)This happens during upload.
There has also been some failed builds at Flathub.
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/168340169
https://flathub.org/builds/#/builders/20/builds/28
The text was updated successfully, but these errors were encountered: