HTTP 429 after HTTP 303 redirect with --retry option retries original request location instead of new location

[kilork]

I did this

POST /report/generate -> 303 Location /poll/<some id>
GET /poll/<some id> -> 429 with Retry-After or 301 with Location /download/<some id>

I am running the following curl command:

curl -v -L --retry 4 -H "Content-Type: application/json" --data '{"hello":"world"}' http://localhost:8000/report/generate

I want to see a single POST request to /report/generate, but instead I am getting first POST request to /report/generate then GET to /poll/..., which returns 429. In this moment it asks for some reason /report/generate, but this time with GET. I think this is a bug.

All logs and also sample server in deno I saved in gist here: https://gist.github.com/kilork/4899caddd6b6f9d9263c0b48c2d36a5c

I expected the following

First request is POST to /report/generate
Then come GET requests to /poll/
And finally I receive 301 to /download/

So basically I want to have poll until done single command. This works correctly with wget, but they do not proceed Retry-After, which also not quite pleasant.

curl/libcurl version

curl 7.70.0 (x86_64-w64-mingw32) libcurl/7.70.0 OpenSSL/1.1.1g (Schannel) zlib/1.2.11 brotli/1.0.7 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS brotli HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink MultiSSL NTLM PSL SPNEGO SSL SSPI TLS-SRP

operating system

Windows 10

[kilork]

This #5237 seems to be also related, there is a long discussion about is it valid case. This issue looks like more complex scenario, where also retry involved.

[jay]

I'd be more interested how we behave in the case of 301 (moved permanently). 303 (see other) is not supposed to be cached and there's no obligation to use the returned URI in future requests.

edit: If 303 then 301 I would think we retry the URI that led to 303. If just 301 I would think we retry the URI in the location header. I'm not 100% on this.

[kilork]

With 301 this was quite similar.

I think of HTTP client in this case as stateless, after 303 it cannot use anything which was not in this 303 request. And 303 is like recommendation to client, so it is not obligated to do anything about that. But in our case it is requesting this location from server, and the answer for this next request - 429. This is already not the original request, it does not make a lot of sense to repeat original request, because 429 is about another location in redirected request.

I would post here logs for 301 also.

[kilork]

Here we go: https://gist.githubusercontent.com/kilork/4899caddd6b6f9d9263c0b48c2d36a5c/raw/5557b6824d58ccd2d2f5c251f2bab0ab24d83e4c/logs%2520if%2520we%2520trigger%2520301

Basically - it is the same.

[jay]

POST /report/generate -> 301 /poll/1
GET /poll/1 -> 429 too many requests
(retry)
GET /report/generate -> 301 /poll/2

Looks like bugs to me. If the original URI should be used on retry then we should send the same request again with the original method. Another issue is when should the original URI be used. For example I would expect this:

POST /report/generate -> 301 /poll/1
GET /poll/1 -> 429 too many requests
(retry)
GET /poll/1 -> 429 too many requests
(retry)
GET /poll/1 -> 429 too many requests
(retry)
GET /poll/1 -> 301 /download/foo
GET /download/foo -> 429 too many requests
(retry)
GET /download/foo -> 429 too many requests
(retry)
GET /download/foo -> 200 ok

[kilork]

Yes, I started with 301, discovered that it does not work, but could also potentionally resend POST, which I do not want, then I red 303 description and was happy, but discovered this also does not work with current curl version. So I ended posting this issue here :) I would be happy with 301 without POST, seems like curl works this way, which also works for most cases. Just it can be an issue when it is not curl.

[bagder]

Looks like bugs to me. If the original URI should be used on retry then we should send the same request again with the original method.

I agree that this seems like a bug - and yes in this case curl should probably spot the redirect and do the retry on the redirected-to URL as I believe that's what this means. The retry isn't actually for the original POST.

[bagder]

With #5499, we change the behavior of libcurl to instead become like this:

POST /report/generate -> 301 /poll/1
GET /poll/1 -> 429 too many requests
(retry)
POST /report/generate -> 301 /poll/1
GET /poll/1 -> 429 too many requests
(retry)

i.e. since curl retries the exact same transfer again, it will do it from the POST. This is also probably not completely what we want, but I think the next step is to make curl (as opposed to libcurl) detect that there's been a POST => GET change and that there's a new URL for the retry so that the retry is done for that GET rather than to redo the POST.

[bagder]

Detecting that in curl is probably not possible without a libcurl feature that can tell what method that was used most recently, and since we're in feature-freeze that can't be merged until next feature window == after the pending release.

So, right now using -L together with --retry will mean that it retries the full thing if the http response code says so.

[kilork]

Thank you for your efforts! I also think it is a good idea to not rush into it.

[bagder]

I'm adding the remainder of this issue to the TODO document and will close this issue.