curl 7.87.0 ftp 1800 files, get 2 errors - ftp timeout, followed by (27) out of memory #10130
Comments
|
What have you done to determine that this is a curl problem and not a problem in git-ftp or in the remote server?
|
|
Too be honest, I have not done anything to rule out git-ftp or the server other than post an issue with git-ftp. Everything was working fine yesterday, then this morning when I updated curl via homebrew, the issue started. That is the only recent change as no other updates have occurred. Git-ftp was lasted updated a year ago. I use cPanel [106.0.11] on my VPS which was updated on Dec 1, 2022 |
|
I just did a quick test by creating 140 files which successfully uploaded. I then reverted the commit and git-ftp successfully deleted the files just uploaded. As best I can determine git-ftp creates a file with the files to upload and passes that to curl. |
|
Without a much more detailed description of what is happening at a protocol level and info about what libcurl is told to do for these transfers and specific info how/when it fails, it is difficult for us to do anything more than blindly guess. |
|
So this morning I did a bit of digging into git-ftp: This is the command that is executed: sample record from the curl_upload_list - 1380 entries The first timeout occurs at record 528 second timeout occurred at record 565 If you can provide some guidance on what else I can do to trouble shoot this issue, I am happy to do so. |
|
NB you've just published your username and password.
|
|
remove now TY, and password reset |
|
There was a fix for an issue where curl would erroneously return "Out of
memory" for a bad URL in yesterday's release, but I don't think that's relevant
in this case, so the "Out of memory" error may very well be legitimate. Have
you tried monitoring memory use of the process when this occurs to see if it's
getting very large?
The "curl: (28) FTP response timeout" error simply looks like the server is bad
or gone down or the connection has gone down.
|
|
I agree the FTP timeout does not appear to be an issue as it does reconnect. I opened the activity monitor and when curl first starts, it is using 2.0MB of memory |
|
It sounds like it's unlikely to be an actual memory issue, but it's still
possible. The OOM error occurs after an attempt at allocating memory has
failed, so it might be trying to allocate a massive memory buffer for some
reason. Does the process have an artificially low max memory set, with ulimit
or cgroups or something? Are you able to instrument curl to find out exactly
where this error 27 is being set before it exits?
|
|
Currently: I don't know what you mean by 'instrument curl' or how to do that. I have been using iterm2 so tried using terminal and got exactly the same results. |
|
2048 means 2 MiB, and you said that when curl return OOM it's using just over
2 MB. That is probably not a coincidence.
|
|
ulimit -u is the number of processes available and not memory. I am well below that in running processes. |
|
Since git-ftp seems to call a separate curl executable, you should be able to
replace it with a special one that logs the location of all the OOM returns so
you can see where this OOM path is hit and see if it's a legitimate OOM or not.
This might be as simple as compiling a debug version of curl and setting the
CURL_MEMDEBUG environment variable when starting git-ftp to a file name to hold
the memory trace file that debug curl creates. This might not work if
git-ftp forks many curl processes in parallel. Alternately, you could replace
the curl executable with a script that starts curl under gdb with breakpoints
on all the OOM return lines (I count 627 of these), or using ltrace on
malloc/calloc/strdup with backtrace enabled to see which one failed.
|
|
git-ftp (bash script) only calls curl once and passes the list of files to upload. I am not a c programmer so don't know how to compile a debug version of curl. If someone could compile a macos version with debug for me, I am happy to run it and upload any logs. I was trying to download a 7.85 macos binary but can't find one to see if that one worked. I really appreciate any help I get. |
|
Once you have all the development tools installed and the curl source code
extracted, you can create a debug curl binary with the commands:
./configure --with-openssl --enable-debug && make -j4
I don't know how you are supposed to run git-ftp, but if you instead run it
something like this:
env PATH="$PWD/src:$PATH" CURL_MEMDEBUG=/tmp/memory-trace.log git-ftp whatever-normally-goes-here
then it will pick up your locally-built curl and enable memory debug logging
into the file /tmp/memory-trace.log. That file should show you the source code
line with the memory allocation that failed, assuming it's a real memory
failure.
|
|
I tried to compile curl with and got the following error" I am running macos Monterey v12.6.2. This way outside my skill set but willing to try again. |
|
That means you need the OpenSSL development package. I'm not sure the best way to install that on Mac OS.
|
|
I couldn't figure out how to compile a debug version of curl so plan B. I was able to restore curl 7.85 from my timemachine backup then reran my original command So I conclude that it is with curl 7.86 & 7.87 and likely the changes in memory handling. |
|
I think it is unlikely to be because of "changes in memory handling" but is more likely a misleading error code. I created a config file with 120 separate upload files in a similar fashion to this report (one line of |
|
Possible it is mac related, so unless someone can provide me with a compiled debug version, I will use 7.85. I can't afford to spend much more time on this. |
|
I just reran my test in my mac with curl from git master. I increased the number of files to see if perhaps the number of uploads could be the key. This time I uploaded 3600 files serially. Each uploaded file was 1.3MB big. Worked fine. I still cannot reproduce this problem. |
|
Thanks @badger, Did you have all the files in a txt file with one call to curl or did you call curl 3600 times? When you say serially, I assume you mean 3800 calls to curl. Also I have curl installed via homebrew. Are these potential differences? |
|
One command line with Serially, as in not parallel ( |
Not to my knowledge, no. |
|
Any chance you could compile a debug version for mac for me and I will try it here. |
|
Sorry, I've not had time for that and I don't think I will for the foreseeable future. |
|
I have had the same issue since late last year as well. I use git-ftp for a side project I don't touch often so it hasn't been worth my time to debug, but getting git-ftp working again would save me a lot of time and hassle, so if I can be of any help as well let me know. I am using macOS 13 Ventura. When I try to use I don't know if the problem started right after moving to Ventura since I upgraded macOS a few months before touching this project again, but I have curl 7.86.0 installed so I am going to try going back to 7.85.0 as well and see if that fixes things for me. |
|
Okay, removing my MacPorts installation of curl 7.86.0 puts me back to the system provided curl 7.85.0, and now I get a different error:
So maybe I was seeing the issue where @dfandrich mentioned with the erroneous error. I'll try 7.87.0 and see what I get.
|
|
Curl 7.87.0 gives me the same output as 7.86.0 did, with I'm not ftp or curl expert, I rarely use them for anything, so any advice on a command to try to manually do what git-ftp is doing for me would be great, so I can see if using curl directly solves the issue. |
|
Argh... I feel like an idiot now, but at least I found my issue. Notice that the file it failed to push had a space in the path. 🤦♂️ Replacing the space with an underscore fixed my issue. That explains why it doesn't matter how many files I tried, that file was always among them, and I changed it late last year, so that explains why I started seeing the issue then. So, the real issue I am seeing is that curl is reporting error |
|
That makes it easy to reproduce: $ curl -T nothing -v 'ftp://example.com/with space'
curl: (27) Out of memoryThat's at HEAD (7.87.1-DEV). |
|
And on the macOS provided 7.85.0 I get: |
|
Now that I fixed that one path issue I installed curl 7.87.0 and tried pushing a larger change (982 files) and I hit the same Going back to 7.85.0 I got the proper |
TODO: There are probably other cases where a CURLUcode should be converted into a CURLcode. Closes #10130
|
Does #10414 fix it for you? |
|
I'll build and try it out. |
|
I am struggling with build settings. I can build curl, but when I try to use it with git-ftp I get errors. Looking at the code changes though I'm pretty confident you got it. Classic C issue where the errors don't get propagated properly. |
|
Actually, all this will fix is to give the ability to git-ftp to display a proper error message. git-ftp will still need to be fixed to escape URLs properly so they are valid when given to curl. |
|
I can confirm that spaces in the path are my issue as well with curl 7.87 |
|
Yes, this doesn't fix the issue with git-ftp, but at least it lets the user know what the actual issue is. And there is a PR open with the fix for git-ftp: git-ftp/git-ftp#638 I applied that change to my local copy and it fixed my issues. |
|
I can confirm that the fix git-ftp/git-ftp#638 fixes the issue for me as well. |
curl would erroneously report CURLE_OUT_OF_MEMORY in some cases instead of CURLE_URL_MALFORMAT. In other cases, it would erroneously return CURLE_URL_MALFORMAT instead of CURLE_OUT_OF_MEMORY. Add a test case to test the former condition. Closes #10130
curl would erroneously report CURLE_OUT_OF_MEMORY in some cases instead of CURLE_URL_MALFORMAT. In other cases, it would erroneously return CURLE_URL_MALFORMAT instead of CURLE_OUT_OF_MEMORY. Add a test case to test the former condition. Fixes curl#10130 Closes curl#10414
I did this
I am using git-ftp v1.6.0 to transfer files from my machine to my VPS.
I am trying to upload 1300 files and delete 500 files
I get the following:
Files start to upload
ftp timeout error
curl continues then fails again with timeout error
Continues a third time then fails with Error(27) Out of memory
I expected the following
all files to upload and delete according to git-ftp transfer log
curl/libcurl version
curl 7.87.0 (x86_64-apple-darwin21.6.0) libcurl/7.87.0 (SecureTransport) OpenSSL/1.1.1s zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libssh2/1.10.0 nghttp2/1.51.0 librtmp/2.3
Release-Date: 2022-12-21
[curl -V output]
curl 7.87.0 (x86_64-apple-darwin21.6.0) libcurl/7.87.0 (SecureTransport) OpenSSL/1.1.1s zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libssh2/1.10.0 nghttp2/1.51.0 librtmp/2.3
Release-Date: 2022-12-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd
operating system
Darwin xxx.local 21.6.0 Darwin Kernel Version 21.6.0: Sun Nov 6 23:31:16 PST 2022; root:xnu-8020.240.14~1/RELEASE_X86_64 x86_64
The text was updated successfully, but these errors were encountered: