curl / curl

Sponsor

Sponsor curl/curl
Watch 704
Star 18.2k
Fork 3.9k

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Honor 429 HTTP Code / Retry-After header #3794

Closed

john-hascall opened this issue Apr 19, 2019 · 9 comments

Labels

HTTP cmdline tool

john-hascall commented Apr 19, 2019

The command line client's --retry option should recognize and honor the 429 HTTP status return and the accompanying 'Retry-After' header.

Affects all versions, but I happen to be running:

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

on:

Linux 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

bagder added cmdline tool HTTP labels

Member

bagder commented Apr 19, 2019

I reasonable suggestion methinks, even if that's a very rarely used response code and header...

Author

john-hascall commented Apr 19, 2019

I think it is relatively common in *Something*-as-a-Service REST APIs. I know that at least three we interact with (Infoblox, Okta, Workday) will return it if we exceed their transactions-per-time-period limits. John

On Fri, Apr 19, 2019 at 10:33 AM Daniel Stenberg ***@***.***> wrote: I reasonable suggestion methinks, even if that's a very rarely used response code and header... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3794 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFTWUXQWSO5H2L2QZ63RPETPRHQ4PANCNFSM4HHEKFXA> .

Member

bagder commented May 3, 2019

I want to address this issue after #3804 has gone in, just to save me from having to do double work.

Contributor

wmark commented Jul 25, 2019

At least one server written in Golang does sent this. It's used when it runs out of file descriptors.

Another sensible use is in anything that runs in the cloud, hits a resource limit (clouds are made of rigid boxes), and cold-start times are known.

Member

bagder commented Aug 6, 2019

It seems basically no browser supports it:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#Browser_compatibility

Author

john-hascall commented Aug 6, 2019

That fits with what I’ve seen, which is a Retry-After header on a HTTP 429 reply from HTTPS REST API servers. I’ve not seen it on the kind of pages you would point a browser at. John John Hascall IT Services Iowa State Univ.

On Aug 6, 2019, at 3:25 AM, Daniel Stenberg ***@***.***> wrote: It seems basically no browser supports it: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#Browser_compatibility — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

bagder added a commit that referenced this issue


          CURLINFO_RETRY_AFTER: parse the Retry-Header value and offer it

602da0d

This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794

bagder mentioned this issue

CURLINFO_RETRY_AFTER: parse the Retry-After header value and offer it #4195

Closed

bagder added a commit that referenced this issue


          CURLINFO_RETRY_AFTER: parse the Retry-After header value

c3123e8

This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794

bagder added a commit that referenced this issue


          CURLINFO_RETRY_AFTER: parse the Retry-After header value

8bf8f0b

This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794

bagder added a commit that referenced this issue


          CURLINFO_RETRY_AFTER: parse the Retry-After header value

This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794

bagder added a commit that referenced this issue


          curl: make use of CURLINFO_RETRY_AFTER when retrying

9564c2d

If a Retry-After: header was used in the response, that value overrides
other retry timing options.

Closes #3794

bagder added a commit that referenced this issue


          CURLINFO_RETRY_AFTER: parse the Retry-After header value

f933449

This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794

bagder closed this in

640b973

Author

john-hascall commented Aug 8, 2019

Thanks, this looks really great. John

On Wed, Aug 7, 2019 at 7:49 AM Daniel Stenberg ***@***.***> wrote: Closed #3794 <#3794> via 640b973 <640b973> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3794?email_source=notifications&email_token=AFTWUXQIUANWULQNNFPN33LQDLAEDA5CNFSM4HHEKFXKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOS5SIRKI#event-2539948201>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFTWUXSSETKHRV5FAPMT4K3QDLAEDANCNFSM4HHEKFXA> .

Contributor

stain commented Oct 2, 2019 •

edited

Thanks @bagder - but when I now try to use this in 7.66.0 against for instance Zenodo API it seems curl is still fails early on 429 HTTP status code - is the retry-logic triggered for 429 just like the 5xx status codes?

Contributor

stain commented Oct 2, 2019

Try for instance

    curl \
    --show-error \
    --fail-early \
    --retry \
    10 \
    --retry-connrefused \
    --location \
    --remote-name-all \
    --dump-header \
    - \
    --header \
    'Accept: application/ld+json' \
    https://zenodo.org/api/records/1299320 \
    https://zenodo.org/api/records/1299783 \
    https://zenodo.org/api/records/1484496 \
    https://zenodo.org/api/records/1308997 \
    https://zenodo.org/api/records/1309962 \
    https://zenodo.org/api/records/1484341 \
    https://zenodo.org/api/records/1310621 \
    https://zenodo.org/api/records/1442453 \
    https://zenodo.org/api/records/1484403 \
    https://zenodo.org/api/records/1421066 \
    https://zenodo.org/api/records/1312623 \
    https://zenodo.org/api/records/1311933 \
    https://zenodo.org/api/records/3343031 \
    https://zenodo.org/api/records/3334688 \
    https://zenodo.org/api/records/3380592 \
    https://zenodo.org/api/records/1484442 \
    https://zenodo.org/api/records/1314672 \
    https://zenodo.org/api/records/3458522 \
    https://zenodo.org/api/records/3445531 \
    https://zenodo.org/api/records/3357806 \
    https://zenodo.org/api/records/3362412 \
    https://zenodo.org/api/records/3458560 \
    https://zenodo.org/api/records/3268809 \
    https://zenodo.org/api/records/1313218 \
    https://zenodo.org/api/records/1312652 \
    https://zenodo.org/api/records/1472385 \
    https://zenodo.org/api/records/1323831 \
    https://zenodo.org/api/records/3336124 \
    https://zenodo.org/api/records/3336148 \
    https://zenodo.org/api/records/1208478 \
    https://zenodo.org/api/records/3338176 \
    https://zenodo.org/api/records/1410052 \
    https://zenodo.org/api/records/3384296 \
    https://zenodo.org/api/records/3270620 \
    https://zenodo.org/api/records/1484480 \
    https://zenodo.org/api/records/3196309 \
    https://zenodo.org/api/records/1484502 \
    https://zenodo.org/api/records/1310034 \
    https://zenodo.org/api/records/1410238 \
    https://zenodo.org/api/records/1408170 \
    https://zenodo.org/api/records/1484424 \
    https://zenodo.org/api/records/1326284 \
    https://zenodo.org/api/records/1473154 \
    https://zenodo.org/api/records/1312685 \
    https://zenodo.org/api/records/1420531 \
    https://zenodo.org/api/records/3459770 \
    https://zenodo.org/api/records/3382423 \
    https://zenodo.org/api/records/1215611 \
    https://zenodo.org/api/records/1311612 \
    https://zenodo.org/api/records/3382259

stain added a commit to stain/curl that referenced this issue


          Ensure HTTP 429 triggers --retry

f294160

This completes curl#3794

Also make sure the new tests from curl#4195 are enabled

stain mentioned this issue

Ensure HTTP 429 triggers --retry #4465

Closed

bagder added a commit that referenced this issue


          curl: ensure HTTP 429 triggers --retry

f7f0b00

This completes #3794.

Also make sure the new tests from #4195 are enabled

Closes #4465

stain mentioned this issue

Log observed time from Retry-After #4498

Closed

lock bot locked as resolved and limited conversation to collaborators

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

You can’t perform that action at this time.