-
-
Notifications
You must be signed in to change notification settings - Fork 6.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
-T -
always uploads with "Transfer-Encoding: chunked"
#12171
Comments
curl will now also compute the content-length of the transfer if stdin is the file to upload and stdin is a regular file, using its file size. Since, while being a regular file, stdin could not have its offset at the start of the file, curl will now also get the current offset into the upload file's file descriptor and use (filesize - offset) as content-length for transfer instead of just using the full filesize. This also fixes a bug on BSDs where open("/dev/fd/N") behaves like dup(N), so, if N is a file descriptor to a regular file, the file offset of the file descriptor returned by open() may not have been at the start of the file despite curl's previous assumption. Since I don't know anything about VMS systems, I left the behaviour for VMS unchanged; on VMS, curl will still perform a chunked transfer if the upload file is stdin. Fixes curl#12171 Fixes curl#12177
curl will now also compute the content-length of the transfer if stdin is the file to upload and stdin is a regular file, using its file size. Since, while being a regular file, stdin could not have its offset at the start of the file, curl will now also get the current offset into the upload file's file descriptor and use (filesize - offset) as content-length for transfer instead of just using the full filesize. This also fixes a bug on BSDs where open("/dev/fd/N") behaves like dup(N), so, if N is a file descriptor to a regular file, the file offset of the file descriptor returned by open() may not have been at the start of the file despite curl's previous assumption. Since I don't know anything about VMS systems, I left the behaviour for VMS unchanged; on VMS, curl will still perform a chunked transfer if the upload file is stdin. Fixes curl#12171 Fixes curl#12177
I'm not following this. curl uses chunked for -T from stdin because it does not know the size beforehand. How is that a bug? |
It does not know its size only because it is not checking it, and there is no reason why it couldn't check it if stdin is a regular file. If you run for example |
Wow, I had no idea it works like this...! 😮 |
curl will now also compute the content-length of the transfer if stdin is the file to upload and stdin is a regular file, using its file size. Since, while being a regular file, stdin could not have its offset at the start of the file, curl will now also get the current offset into the upload file's file descriptor and use (filesize - offset) as content-length for transfer instead of just using the full filesize. This also fixes a bug on BSDs where open("/dev/fd/N") behaves like dup(N), so, if N is a file descriptor to a regular file, the file offset of the file descriptor returned by open() may not have been at the start of the file despite curl's previous assumption. Since I don't know anything about VMS systems, I left the behaviour for VMS unchanged; on VMS, curl will still perform a chunked transfer if the upload file is stdin. I updated tests that were testing chunked transfers for stdin to use <stdin pipe="yes"> since now curl only performs a chunked transfer by default if the upload file (stdin or not) is not a regular file. Fixes curl#12171 Fixes curl#12177
curl will now also compute the content-length of the transfer if stdin is the file to upload and stdin is a regular file, using its file size. Since, while being a regular file, stdin could not have its offset at the start of the file, curl will now also get the current offset into the upload file's file descriptor and use (filesize - offset) as content-length for transfer instead of just using the full filesize. This also fixes a bug on BSDs where open("/dev/fd/N") behaves like dup(N), so, if N is a file descriptor to a regular file, the file offset of the file descriptor returned by open() may not have been at the start of the file despite curl's previous assumption. Since I don't know anything about VMS systems, I left the behaviour for VMS unchanged; on VMS, curl will still perform a chunked transfer if the upload file is stdin. I updated tests that were testing chunked transfers for stdin to use <stdin pipe="yes"> since now curl only performs a chunked transfer by default if the upload file (stdin or not) is not a regular file. Fixes curl#12171 Fixes curl#12177
curl will now also compute the content-length of the transfer if stdin is the file to upload and stdin is a regular file, using its file size. Since, while being a regular file, stdin could not have its offset at the start of the file, curl will now also get the current offset into the upload file's file descriptor and use (filesize - offset) as content-length for transfer instead of just using the full filesize. This also fixes a bug on BSDs where open("/dev/fd/N") behaves like dup(N), so, if N is a file descriptor to a regular file, the file offset of the file descriptor returned by open() may not have been at the start of the file despite curl's previous assumption. Since I don't know anything about VMS systems, I left the behaviour for VMS unchanged; on VMS, curl will still perform a chunked transfer if the upload file is stdin. I updated tests that were testing chunked transfers for stdin to use <stdin pipe="yes"> since now curl only performs a chunked transfer by default if the upload file (stdin or not) is not a regular file. Fixes curl#12171 Fixes curl#12177
curl will now also compute the content-length of the transfer if stdin is the file to upload and stdin is a regular file, using its file size. Since, while being a regular file, stdin could not have its offset at the start of the file, curl will now also get the current offset into the upload file's file descriptor and use (filesize - offset) as content-length for transfer instead of just using the full filesize. This also fixes a bug on BSDs where open("/dev/fd/N") behaves like dup(N), so, if N is a file descriptor to a regular file, the file offset of the file descriptor returned by open() may not have been at the start of the file despite curl's previous assumption. Since I don't know anything about VMS systems, I left the behaviour for VMS unchanged; on VMS, curl will still perform a chunked transfer if the upload file is stdin. I updated tests that were testing chunked transfers for stdin to use <stdin pipe="yes"> since now curl only performs a chunked transfer by default if the upload file (stdin or not) is not a regular file. And I added a new test (258) that verifies that stdin is uploaded with content-length (not transfer-encoding:chunked) if it is a regular file. Fixes curl#12171 Fixes curl#12177
curl will now also compute the content-length of the transfer if stdin is the file to upload and stdin is a regular file, using its file size. Since, while being a regular file, stdin could not have its offset at the start of the file, curl will now also get the current offset into the upload file's file descriptor and use (filesize - offset) as content-length for transfer instead of just using the full filesize. This also fixes a bug on BSDs where open("/dev/fd/N") behaves like dup(N), so, if N is a file descriptor to a regular file, the file offset of the file descriptor returned by open() may not have been at the start of the file despite curl's previous assumption. Since I don't know anything about VMS systems, I left the behaviour for VMS unchanged; on VMS, curl will still perform a chunked transfer if the upload file is stdin. I updated tests that were testing chunked transfers for stdin to use <stdin pipe="yes"> since now curl only performs a chunked transfer by default if the upload file (stdin or not) is not a regular file. And I added a new test (268) that verifies that stdin is uploaded with content-length (not transfer-encoding:chunked) if it is a regular file. Fixes curl#12171 Fixes curl#12177
I did this
I expected the following
curl
should realise that stdin is a regular file, and that it can do a non-chunked upload, like it would do if you used-T ~/.bashrc
.(EDIT: Actually, that is not true; I was misremembering;curl
already does that when you use@-
or<-
with-F
/-d
and similar, but not when you use-T
.curl
only knows the content length for stdin with-F
/-d
because it always loads the entire file into memory before sending it)The problem is that
curl -T
never stats stdin to figure if it is a regular file when you use-
as argument.On Linux and BSDs, a possible workround is
-T /dev/stdin
(with--request-target /
); since a path was provided,curl
will stat it and realise it is a regular file, but it would be nicer ifcurl
would just do that on its own.I wanted to create a patch to fix this bug when I first learned about it, but the code that deals with stat-ing the file for upload (in
src/tool_operate.c
) is doing something special on#define __VMS
systems to figure out the real upload size of regular files that involes opening the file, and reading all its data to count bytes. I didn't know why it did that, or how to do that for stdin if the system is VMS (maybe usinglseek
to go to the start of stdin), and anyway I would not be able to test the code I write for VMS on VMS, so I gave up trying to write the patch myself to avoid accidentally breaking the VMS port.I remembered about this bug today, and noticed no one created an issue for it.
Ref: #11396 (comment)
curl/libcurl version
curl 8.4.0
operating system
Arch Linux
The text was updated successfully, but these errors were encountered: