Honor 429 HTTP Code / Retry-After header

[john-hascall]

The command line client's --retry option should recognize and honor the 429 HTTP status return and the accompanying 'Retry-After' header.

Affects all versions, but I happen to be running:

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

on:

Linux 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[bagder]

I reasonable suggestion methinks, even if that's a very rarely used response code and header...

[john-hascall]

I think it is relatively common in *Something*-as-a-Service REST APIs. I know that at least three we interact with (Infoblox, Okta, Workday) will return it if we exceed their transactions-per-time-period limits. John

…

On Fri, Apr 19, 2019 at 10:33 AM Daniel Stenberg ***@***.***> wrote: I reasonable suggestion methinks, even if that's a very rarely used response code and header... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3794 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFTWUXQWSO5H2L2QZ63RPETPRHQ4PANCNFSM4HHEKFXA> .

[bagder]

I want to address this issue after #3804 has gone in, just to save me from having to do double work.

[wmark]

At least one server written in Golang does sent this. It's used when it runs out of file descriptors.

Another sensible use is in anything that runs in the cloud, hits a resource limit (clouds are made of rigid boxes), and cold-start times are known.

[bagder]

It seems basically no browser supports it:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#Browser_compatibility

[john-hascall]

That fits with what I’ve seen, which is a Retry-After header on a HTTP 429 reply from HTTPS REST API servers. I’ve not seen it on the kind of pages you would point a browser at. John John Hascall IT Services Iowa State Univ.

…

On Aug 6, 2019, at 3:25 AM, Daniel Stenberg ***@***.***> wrote: It seems basically no browser supports it: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#Browser_compatibility — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[john-hascall]

Thanks, this looks really great. John

…

On Wed, Aug 7, 2019 at 7:49 AM Daniel Stenberg ***@***.***> wrote: Closed #3794 <#3794> via 640b973 <640b973> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3794?email_source=notifications&email_token=AFTWUXQIUANWULQNNFPN33LQDLAEDA5CNFSM4HHEKFXKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOS5SIRKI#event-2539948201>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFTWUXSSETKHRV5FAPMT4K3QDLAEDANCNFSM4HHEKFXA> .

[stain]

Thanks @bagder - but when I now try to use this in 7.66.0 against for instance Zenodo API it seems curl is still fails early on 429 HTTP status code - is the retry-logic triggered for 429 just like the 5xx status codes?

[stain]

Try for instance

    curl \
    --show-error \
    --fail-early \
    --retry \
    10 \
    --retry-connrefused \
    --location \
    --remote-name-all \
    --dump-header \
    - \
    --header \
    'Accept: application/ld+json' \
    https://zenodo.org/api/records/1299320 \
    https://zenodo.org/api/records/1299783 \
    https://zenodo.org/api/records/1484496 \
    https://zenodo.org/api/records/1308997 \
    https://zenodo.org/api/records/1309962 \
    https://zenodo.org/api/records/1484341 \
    https://zenodo.org/api/records/1310621 \
    https://zenodo.org/api/records/1442453 \
    https://zenodo.org/api/records/1484403 \
    https://zenodo.org/api/records/1421066 \
    https://zenodo.org/api/records/1312623 \
    https://zenodo.org/api/records/1311933 \
    https://zenodo.org/api/records/3343031 \
    https://zenodo.org/api/records/3334688 \
    https://zenodo.org/api/records/3380592 \
    https://zenodo.org/api/records/1484442 \
    https://zenodo.org/api/records/1314672 \
    https://zenodo.org/api/records/3458522 \
    https://zenodo.org/api/records/3445531 \
    https://zenodo.org/api/records/3357806 \
    https://zenodo.org/api/records/3362412 \
    https://zenodo.org/api/records/3458560 \
    https://zenodo.org/api/records/3268809 \
    https://zenodo.org/api/records/1313218 \
    https://zenodo.org/api/records/1312652 \
    https://zenodo.org/api/records/1472385 \
    https://zenodo.org/api/records/1323831 \
    https://zenodo.org/api/records/3336124 \
    https://zenodo.org/api/records/3336148 \
    https://zenodo.org/api/records/1208478 \
    https://zenodo.org/api/records/3338176 \
    https://zenodo.org/api/records/1410052 \
    https://zenodo.org/api/records/3384296 \
    https://zenodo.org/api/records/3270620 \
    https://zenodo.org/api/records/1484480 \
    https://zenodo.org/api/records/3196309 \
    https://zenodo.org/api/records/1484502 \
    https://zenodo.org/api/records/1310034 \
    https://zenodo.org/api/records/1410238 \
    https://zenodo.org/api/records/1408170 \
    https://zenodo.org/api/records/1484424 \
    https://zenodo.org/api/records/1326284 \
    https://zenodo.org/api/records/1473154 \
    https://zenodo.org/api/records/1312685 \
    https://zenodo.org/api/records/1420531 \
    https://zenodo.org/api/records/3459770 \
    https://zenodo.org/api/records/3382423 \
    https://zenodo.org/api/records/1215611 \
    https://zenodo.org/api/records/1311612 \
    https://zenodo.org/api/records/3382259