Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'Server returned HTTP 503' from time to time. #845

Closed
mkrajnak opened this issue Feb 6, 2019 · 30 comments
Closed

'Server returned HTTP 503' from time to time. #845

mkrajnak opened this issue Feb 6, 2019 · 30 comments

Comments

@mkrajnak
Copy link

mkrajnak commented Feb 6, 2019

I noticed that when I am using flathub frequently, e.g. installing several apps in a row i've have been getting this error from time to time with some apps (not the same app), in both command line and gnome-software.

index

error: Failed to install org.jamovi.jamovi/x86_64/stable: While pulling app/org.jamovi.jamovi/x86_64/stable from remote flathub: Server returned HTTP 503\n'

This might be rather a question than issue, so is this a security precaution from the server to prevent an overload from certain clients or am I hitting some possible issue with service stability ?

@mcatanzaro
Copy link

Seems to be impossible to install org.gnome.Sdk.Debug//3.30 right now. I've tried three times:

 flatpak install --user org.gnome.Sdk//3.30 org.gnome.Sdk.Debug//3.30
Looking for matches…
Found similar ref(s) for ‘org.gnome.Sdk//3.30’ in remote ‘flathub’ (user).
Use this remote? [Y/n]: y


        ID                     Arch     Branch   Remote    Download
 1. [✗] org.gnome.Sdk.Debug    x86_64   3.30     flathub     196.1 MB / 6.2 GB
 2. [ ] org.gnome.Sdk          x86_64   3.30     flathub     < 1.8 GB
 3. [ ] org.gnome.Sdk.Locale   x86_64   3.30     flathub   < 226.3 MB (partial)

Error: While pulling runtime/org.gnome.Sdk.Debug/x86_64/3.30 from remote flathub: Server returned HTTP 503
error: Failed to install org.gnome.Sdk.Debug: While pulling runtime/org.gnome.Sdk.Debug/x86_64/3.30 from remote flathub: Server returned HTTP 503

@rugk
Copy link

rugk commented Apr 1, 2019

Note that it really only happens for this one flatpak then for some time. So it really looks like a DDOS protection… (but you do have to expect many and large file downloads, that's just how it is)

@mkrajnak
Copy link
Author

mkrajnak commented Apr 2, 2019

Note that it really only happens for this one flatpak then for some time. So it really looks like a DDOS protection… (but you do have to expect many and large file downloads, that's just how it is)

Well this is happening for me for multiple flatpaks from flathub not only one :(

@rugk
Copy link

rugk commented Apr 2, 2019

Yes, but not all of them, so… that was my point.

@pouar
Copy link

pouar commented Dec 31, 2019

Happens every time I try to install org.freedesktop.Sdk.Extension.texlive. Wouldn't be so bad if flatpak didn't try to redownload the whole thing from scratch instead of resuming where it left off.

@pouar
Copy link

pouar commented Dec 31, 2019

There wouldn't happen to be a way to get flatpak to resume from where it left off that I don't know about would there?

@forember
Copy link

I too am having this issue. How long do you usually have to wait?

@mkrajnak
Copy link
Author

Not sure but since more and people are having trouble with this I suspect that flathub has probably some problems dealing with heavy traffic.

@barthalion
Copy link
Member

I'm trying to free some disk space as publishing updates is blocked for 20-30h now and that kills the drive. I applied some ionice but can't promise how much it's going to help. Going to stop the process in 2-3 hours.

@barthalion
Copy link
Member

OK, we're back. I'll close it for now; next week we're migrating to new server and that should help with infrastructure-related 503s.

@barthalion
Copy link
Member

Migration was finished so hopefully random 503 will be gone.

@junjieyuan
Copy link

This issue may still not be resolved. Downloading large files is extremely slow and still returns 503.

I downloaded texstudio twice, the download speed is very low (<45 KB/s), and I always get 503 errors.

texstudio issue

[junjie@xps ~]$ flatpak install flathub org.texstudio.TeXstudio
Looking for matches…
Required runtime for org.texstudio.TeXstudio/x86_64/stable (runtime/org.kde.Platform/x86_64/5.12) found in remote flathub
Do you want to install it? [Y/n]: 

org.texstudio.TeXstudio permissions:
    ipc      network      x11      dri     file access [1]     dbus access [2]

    [1] host, xdg-config/kdeglobals:ro, xdg-run/dconf, ~/.config/dconf:ro
    [2] com.canonical.AppMenu.Registrar


        ID                                                             Arch     Branch   Remote    Download
 1. [✓] org.kde.Platform                                               x86_64   5.12     flathub   367.0 MB / 430.1 MB
 2. [✓] org.freedesktop.Platform.VAAPI.Intel                           x86_64   18.08    flathub     1.8 MB / 1.8 MB
 3. [✓] org.freedesktop.Platform.html5-codecs                          x86_64   18.08    flathub     3.0 MB / 3.3 MB
 4. [✓] org.kde.KStyle.Adwaita                                         x86_64   5.12     flathub     5.6 MB / 5.6 MB
 5. [✓] org.kde.PlatformTheme.QGnomePlatform.Sources                   x86_64   5.12     flathub    30.9 kB / 29.7 kB
 6. [✓] org.kde.PlatformTheme.QGnomePlatform                           x86_64   5.12     flathub   145.1 kB / 144.1 kB
 7. [✓] org.kde.WaylandDecoration.QGnomePlatform-decoration            x86_64   5.12     flathub    59.9 kB / 58.5 kB
 8. [✓] org.kde.WaylandDecoration.QGnomePlatform_decoration.Sources    x86_64   5.12     flathub     1.8 kB / 29.8 kB
 9. [✓] org.kde.Platform.Locale                                        x86_64   5.12     flathub    17.3 kB / 334.3 MB
10. [✗] org.texstudio.TeXstudio                                        x86_64   stable   flathub   446.4 MB / 1.9 GB

Error: While pulling app/org.texstudio.TeXstudio/x86_64/stable from remote flathub: Server returned HTTP 503
error: Failed to install org.texstudio.TeXstudio: While pulling app/org.texstudio.TeXstudio/x86_64/stable from remote flathub: Server returned HTTP 503

@mkrajnak
Copy link
Author

Same old,

Looking for matches…
Required runtime for org.gnome.Eolie/x86_64/stable (runtime/org.gnome.Platform/x86_64/3.34) found in remote flathub

org.gnome.Eolie permissions:
    ipc	network	pulseaudio	wayland
    x11	dri	file access [1]	dbus access [2]
    bus ownership [3]

    [1] home, ~/.config/chrome, ~/.config/chromium, ~/.mozilla/firefox
    [2] org.freedesktop.secrets
    [3] org.gnome.Eolie.Proxy.*


 1.	   	org.freedesktop.Platform.GL.default	19.08	i	flathub	< 92.6 MB
 2.	   	org.freedesktop.Platform.openh264	19.08	i	flathub	< 593.4 kB
 3.	   	org.gnome.Eolie.Locale	stable	i	flathub	< 222.0 kB (partial)
 4.	   	org.gnome.Platform.Locale	3.34	i	flathub	< 322.2 MB (partial)
 5.	   	org.gnome.Platform	3.34	i	flathub	< 318.6 MB
 6.	   	org.gnome.Eolie	stable	i	flathub	< 12.9 MB



Installing 1/6…                                                                   
Installing 1/6… ▍                      2%
Installing 1/6… █████████▏            46%  2.3 MB/s
Installing 1/6… █████████▊            49%  3.7 MB/s
Installing 1/6… ██████████            50%  5.1 MB/s
Installing 1/6… ██████████▍           52%  6.9 MB/s
Installing 1/6… ███████████           55%  4.2 MB/s
Installing 1/6… ███████████▏          56%  4.9 MB/s
Installing 1/6… ███████████▍          57%  5.3 MB/s
Installing 1/6… ███████████▌          58%  3.9 MB/s
Installing 1/6… ███████████▊          59%  4.2 MB/s
Installing 1/6… ████████████          60%  4.6 MB/s
Installing 1/6… ████████████▍         62%  3.9 MB/s  00:02
Installing 1/6… ████████████▌         63%  4.2 MB/s  00:02
Installing 1/6… ████████████▌         63%  4.5 MB/s  00:02
Installing 1/6… ████████████▌         63%  4.9 MB/s  00:02
Installing 1/6… ████████████▌         63%  4.2 MB/s  00:02
Installing 1/6… ████████████▌         63%  4.4 MB/s  00:02
Installing 1/6… ████████████▌         63%  4.5 MB/s  00:02
Installing 1/6… ████████████▌         63%  3.9 MB/s  00:03
Installing 1/6… ████████████▊         64%  4.1 MB/s  00:03
Installing 1/6… █████████████         65%  4.3 MB/s  00:03
Installing 1/6… █████████████         65%  3.8 MB/s  00:03
Installing 1/6… █████████████         65%  3.8 MB/s  00:03
Installing 1/6… █████████████▏        66%  3.9 MB/s  00:03
Installing 1/6… █████████████▏        66%  4.0 MB/s  00:03
Installing 1/6… █████████████▏        66%  3.6 MB/s  00:04
Installing 1/6… █████████████▏        66%  3.7 MB/s  00:04
Installing 1/6… █████████████▏        66%  3.7 MB/s  00:04
Installing 1/6… █████████████▍        67%  3.4 MB/s  00:04
Installing 1/6… █████████████▍        67%  3.5 MB/s  00:04
Installing 1/6… █████████████▌        68%  3.6 MB/s  00:04
Installing 1/6… █████████████▌        68%  3.4 MB/s  00:04
Installing 1/6… █████████████▌        68%  3.5 MB/s  00:04
Installing 1/6… █████████████▊        69%  3.5 MB/s  00:04
Installing 1/6… █████████████▊        69%  3.3 MB/s  00:04
Installing 1/6… ██████████████        70%  3.4 MB/s  00:04
Installing 1/6… ██████████████▍       72%  3.5 MB/s  00:04
Installing 1/6… ██████████████▌       73%  3.6 MB/s  00:04
Installing 1/6… ██████████████▊       74%  3.4 MB/s  00:04
Installing 1/6… ███████████████       75%  3.5 MB/s  00:04
Installing 1/6… ███████████████▏      76%  3.6 MB/s  00:03
Installing 1/6… ███████████████▌      78%  3.5 MB/s  00:03
Installing 1/6… ███████████████▌      78%  3.5 MB/s  00:03
Installing 1/6… ███████████████▊      79%  3.6 MB/s  00:03
Installing 1/6… ████████████████      80%  3.4 MB/s  00:03
Installing 1/6… ████████████████▏     81%  3.4 MB/s  00:03
Installing 1/6… ████████████████▍     82%  3.5 MB/s  00:03
Installing 1/6… ████████████████▊     84%  3.6 MB/s  00:02
Installing 1/6… █████████████████▏    86%  3.4 MB/s  00:02
Installing 1/6… █████████████████▍    87%  3.5 MB/s  00:02
Installing 1/6… █████████████████▌    88%  3.6 MB/s  00:02
Installing 1/6… ██████████████████    90%  3.5 MB/s  00:01
Installing 1/6… ██████████████████▏   91%  3.6 MB/s  00:01
Installing 1/6… ██████████████████▍   92%  3.7 MB/s  00:01
Installing 1/6… ██████████████████▌   93%  3.5 MB/s  00:01
Installing 1/6… ██████████████████▊   94%  3.6 MB/s  00:01
Installing 1/6… ██████████████████▊   94%  3.6 MB/s  00:01
Installing 1/6… ██████████████████▊   94%  3.7 MB/s  00:01
Installing 1/6… ███████████████████▏  96%  3.5 MB/s  00:00
Installing 1/6… ███████████████████▏  96%  3.6 MB/s  00:00
Installing 1/6… ███████████████████▏  96%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▏  96%  3.6 MB/s  00:00
Installing 1/6… ███████████████████▏  96%  3.6 MB/s  00:00
Installing 1/6… ███████████████████▏  96%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▏  96%  3.5 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.6 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.6 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.6 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.8 MB/s  00:00
Installing 1/6… ███████████████████▍  97%  3.8 MB/s  00:00
Installing 1/6… ███████████████████▌  98%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▌  98%  3.8 MB/s  00:00
Installing 1/6… ███████████████████▌  98%  3.8 MB/s  00:00
Installing 1/6… ███████████████████▌  98%  3.8 MB/s  00:00
Installing 1/6… ███████████████████▌  98%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▌  98%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▊  99%  3.8 MB/s  00:00
Installing 1/6… ███████████████████▊  99%  3.6 MB/s  00:00
Installing 1/6… ███████████████████▊  99%  3.7 MB/s  00:00
Installing 1/6… ███████████████████▊  99%  3.7 MB/s  00:00
Installing 1/6… ████████████████████ 100%  3.6 MB/s  00:00
Installing 2/6…                                                                   
Installing 2/6… █▌                     8%
Installing 2/6… ████████████████████ 100%
Installing 2/6… ████████████████████ 100%
Installing 3/6…                                                                   
Installing 3/6…                        0%
Installing 3/6… ████████████████████ 100%
Installing 4/6…                                                                   
Installing 4/6… ▍                      2%
Installing 4/6… ████████████████████ 100%
Installing 5/6…                                                                   
Installing 5/6… ██▍                   12%
Installing 5/6… ██▏                   11%
Installing 5/6… ██▏                   11%  495.0 kB/s
Installing 5/6… ██▏                   11%  582.1 kB/s
Installing 5/6… ██▏                   11%  702.6 kB/s
Installing 5/6… ██▏                   11%  366.1 kB/s
Installing 5/6… ██▏                   11%  372.3 kB/s
Installing 5/6… ██▏                   11%  248.5 kB/s
Installing 5/6… ██▏                   11%  248.9 kB/s
Err:
Warning: org.freedesktop.Platform.openh264 not installed                            

Error: While pulling runtime/org.gnome.Platform/x86_64/3.34 from remote flathub: Server returned HTTP 503
error: Failed to install org.gnome.Platform: While pulling runtime/org.gnome.Platform/x86_64/3.34 from remote flathub: Server returned HTTP 503

@jonathon-love
Copy link

hi,

would it be possible to reopen this issue?

i occasionally have to install my own package from flathub, and am often surprise how many times i have to retry it, before i can get it to install. i was pretty glad when you guys mentioned new infrastructure, but this doesn't seem to have made a difference (in my limited experience).

i'm org.jamovi.jamovi if you'd like to try, it sounds like some (larger?) packages are impacted more than others.

with thanks

@dbarnett
Copy link

dbarnett commented Mar 7, 2020

Possibly related to same root cause as #505, where it may have been returning 404 responses when it's overloaded.

Definitely unresolved, and a major usability issue with flathub and flatpak. Would love to see the issue at least acknowledged.

@jonathon-love
Copy link

i have found this to be very useful when installing things from flathub, in fact it should probably go in the installation instructions.

until flatpak install org.blah.Blah; do echo "Trying again"; sleep 2; done

you'll notice that between 503's, progress is actually being made, so this isn't redownloading the same content over and over.

cheers

@alexlarsson
Copy link
Member

I think we should figure out the exact cause of this, but also i filed ostreedev/ostree#2022 about how ostree could auto-detect this and retry a few times.

@alexlarsson alexlarsson reopened this Mar 9, 2020
@barthalion
Copy link
Member

I don't see any 503 errors on our load balancers. Around two hundred 503 errors were reported by Fastly to our stats box, but I don't think it's the issue we're talking about here as it sounds like something more common.

@jonathon-love
Copy link

i don't know if this is related, but downloading test builds is remarkably slow too. it's quite typical to experience downloads at about 30k/s. combined with the 503s, a test download can take 3-5 hours to download.

@sudden6
Copy link

sudden6 commented Mar 14, 2020

I'm facing the same 503 error on our Travis CI setup when downloading runtime/org.kde.Sdk/x86_64/5.11

sudden6 added a commit to sudden6/qTox that referenced this issue Mar 14, 2020
sudden6 added a commit to sudden6/qTox that referenced this issue Mar 14, 2020
sudden6 added a commit to sudden6/qTox that referenced this issue Mar 14, 2020
sudden6 added a commit to sudden6/qTox that referenced this issue Mar 14, 2020
sudden6 added a commit to sudden6/qTox that referenced this issue Mar 14, 2020
sudden6 added a commit to sudden6/qTox that referenced this issue Mar 14, 2020
@jonathon-love
Copy link

@sudden6, you may actually find the 503's helpful on travis,

travis has a 50 minute job limit, and downloading a flatpak can take much longer than that.

if you're not running up against the time limit then just use:

until flatpak install org.blah.Blah; do echo "Trying again"; sleep 2; done

if you're running up against the time limit, the 503's will actually help you out. just list lots of installs like this:

 - flatpak install --user -y org.blah.Blah || true
 - flatpak install --user -y org.blah.Blah || true
 - flatpak install --user -y org.blah.Blah || true
 - flatpak install --user -y org.blah.Blah || true
 # 20-30 more, depending on how big your download is

 # then place this at the end
 - until flatpak install --user -y org.blah.Blah; do echo "Trying again"; sleep 2; done

by distributing the download over 30 separate 'jobs', no single job takes more than 50 minutes (where as if you just use a single until that counts as a single job). i honestly don't know how i'm going to install things on travis once they fix the 503's issue :P

once you've got it downloaded to travis, cache that sucker!

cache:
  directories:
    - $HOME/.local/share/flatpak/app/

and then have a script that only downloads if it's not in the cache:

if flatpak info --user org.blah.Blah 1>/dev/null 2>/dev/null ; then
    echo "already installed"
else
    echo "installing"
    flatpak install -y --user flathub org.blah.Blah
fi

i've found this works quite well.

cheers

@sudden6
Copy link

sudden6 commented Mar 15, 2020

@jonathon-love Thank you I arrived at the same conclusion, that downloading multiple times is probably the solution.

@barthalion
Copy link
Member

@sudden6 You should also consider upgrading flatpak to latest 1.4.x or 1.6.x releases. In previous releases there is a regression causing flatpak not to use static deltas.

@sisterfelicity
Copy link

I'm having this same error trying to install flatpak MuseScore -- see the thread I started here: @flathub/org.musescore.MuseScore#53

@jonathon-love, I'd like to try your "trying again" loop workaround, but I'm clearly getting it wrong ... please help with idiot-proof syntax? Thanks!

image

@jonathon-love
Copy link

ah yup, my bad, you're missing the repo (in this case flathub) in your call to flatpak install.

try this:

until flatpak install -y flathub org.musescore.MuseScore; do echo "Trying again"; sleep 2; done

@sisterfelicity
Copy link

Thanks -- I tried it with and without prefixing "sudo", still missing something:

Screenshot 2020-03-29 at 05 39 24

Tried replacing "do" with "sudo", and got the same syntax error but near 'done' instead of near 'do'

@jonathon-love
Copy link

hmm, i'm slightly at a loss. that command works for me:

Screen Shot 2020-03-29 at 20 50 24

might have to await someone else's input.

@sisterfelicity
Copy link

OK, thanks again. Patience is a virtue, they tell me.

@dbarnett
Copy link

FWIW, we've digressed several levels from why the server craps out sometimes to troubleshooting a particular invocation scenario of a hacky retry script.

Flathub devs, please consider putting some more attention into fixing the actual 503 problem. If there is a way, it'd be win-win: I suspect having people loop and retry will also not be kind to your servers.

@barthalion
Copy link
Member

The issue is gone now. Nodes in our load balancer were not keeping up with traffic from CDN; they were replaced with new SSD-backed hosts and number of 503s in CDN logs is down to zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests