curl 7.87.0 ftp 1800 files, get 2 errors - ftp timeout, followed by (27) out of memory

[kitsguru]

I did this

I am using git-ftp v1.6.0 to transfer files from my machine to my VPS.
I am trying to upload 1300 files and delete 500 files
I get the following:
Files start to upload
ftp timeout error
curl continues then fails again with timeout error
Continues a third time then fails with Error(27) Out of memory

I expected the following

all files to upload and delete according to git-ftp transfer log

curl/libcurl version

curl 7.87.0 (x86_64-apple-darwin21.6.0) libcurl/7.87.0 (SecureTransport) OpenSSL/1.1.1s zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libssh2/1.10.0 nghttp2/1.51.0 librtmp/2.3
Release-Date: 2022-12-21

[curl -V output]
curl 7.87.0 (x86_64-apple-darwin21.6.0) libcurl/7.87.0 (SecureTransport) OpenSSL/1.1.1s zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libssh2/1.10.0 nghttp2/1.51.0 librtmp/2.3
Release-Date: 2022-12-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

operating system

Darwin xxx.local 21.6.0 Darwin Kernel Version 21.6.0: Sun Nov 6 23:31:16 PST 2022; root:xnu-8020.240.14~1/RELEASE_X86_64 x86_64

[dfandrich]

What have you done to determine that this is a curl problem and not a problem in git-ftp or in the remote server?

[kitsguru]

Too be honest, I have not done anything to rule out git-ftp or the server other than post an issue with git-ftp.

Everything was working fine yesterday, then this morning when I updated curl via homebrew, the issue started. That is the only recent change as no other updates have occurred.

Git-ftp was lasted updated a year ago. I use cPanel [106.0.11] on my VPS which was updated on Dec 1, 2022

[kitsguru]

I just did a quick test by creating 140 files which successfully uploaded. I then reverted the commit and git-ftp successfully deleted the files just uploaded.

As best I can determine git-ftp creates a file with the files to upload and passes that to curl.

[bagder]

Without a much more detailed description of what is happening at a protocol level and info about what libcurl is told to do for these transfers and specific info how/when it fails, it is difficult for us to do anything more than blindly guess.

[kitsguru]

So this morning I did a bit of digging into git-ftp:

This is the command that is executed:

curl -s --globoff -# --ftp-create-dirs -K /var/folders/p1/2q6_xz196qsg9g3j67d8qf7c0000gn/T/git-ftp-XXXXXX.C12jHSJ5/curl_upload_list

sample record from the curl_upload_list - 1380 entries
xxxxx is the domain
yyyyyyy was the password

-T "./www/templates/site/xxxxx/right-col.twig"
url = "ftp://xxxxx:yyyyyyy@ftp.xxxxx.com/www/templates/site/xxxxx/right-col.twig"

The first timeout occurs at record 528

* Remembering we are in dir "www/cmsb/lib//menus/_codeGenerator/"
< 226-File successfully transferred
< 226 0.083 seconds (measured here), 52.88 Kbytes per second
* Connection #0 to host ftp.xxxxx.com left intact

* Found bundle for host: 0x600003faf540 [serially]
* Re-using existing connection #0 with host ftp.xxxxx.com
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Extended Passive mode OK (|||57903|)
* Connecting to 111.111.111.111 (111.111.111.111) port 57903
*   Trying 111.111.111.111:57903...
* Connected to ftp.xxxxx.com (111.111.111.111) port 21 (#0)
> STOR api_categoryMenu.php
< 150 Accepted data connection
} [15151 bytes data]
* We are completely uploaded and fine
################################################################################################################# 100.0%
* Remembering we are in dir "www/cmsb/lib//menus/_codeGenerator/"
* FTP response timeout
* control connection looks dead
* Closing connection 0
curl: (28) FTP response timeout

*   Trying 111.111.111.111:21...
* Connected to ftp.xxxxx.com (111.111.111.111) port 21 (#1)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 1 of 50 allowed.
< 220-Local time is now 16:44. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.

< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD www
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /public_html
> CWD cmsb
< 250 OK. Current directory is /public_html/cmsb
> CWD lib
< 250 OK. Current directory is /public_html/cmsb/lib
> CWD menus
< 250 OK. Current directory is /public_html/cmsb/lib/menus
> CWD _codeGenerator
< 250 OK. Current directory is /public_html/cmsb/lib/menus/_codeGenerator
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||49664|)
* Connecting to 111.111.111.111 (111.111.111.111) port 49664
*   Trying 111.111.111.111:49664...
* Connected to ftp.xxxxx.com (111.111.111.111) port 21 (#1)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR api_comboPage.php
< 150 Accepted data connection
} [9745 bytes data]
* We are completely uploaded and fine

second timeout occurred at record 565
died on record 636

* Remembering we are in dir "www/cmsb/plugins//csvExport/"
< 226-File successfully transferred
< 226 0.084 seconds (measured here), 23.43 Kbytes per second
* Connection #2 to host ftp.xxxxx.com left intact

* Found bundle for host: 0x600003fa0060 [serially]
* Re-using existing connection #2 with host ftp.xxxxx.com
* Request has same path as previous transfer
> EPSV
* Connect data stream passively
* ftp_perform ends with SECONDARY: 0
< 229 Extended Passive mode OK (|||63906|)
* Connecting to 111.111.111.111 (111.111.111.111) port 63906
*   Trying 111.111.111.111:63906...
* Connected to ftp.xxxxx.com (111.111.111.111) port 21 (#2)
> STOR csvExport.php
< 150 Accepted data connection
} [9432 bytes data]
* We are completely uploaded and fine
################################################################################################################# 100.0%
* Remembering we are in dir "www/cmsb/plugins//csvExport/"
< 226-File successfully transferred
< 226 0.083 seconds (measured here), 111.52 Kbytes per second
* Connection #2 to host ftp.xxxxx.com left intact

curl: (27) Out of memory

If you can provide some guidance on what else I can do to trouble shoot this issue, I am happy to do so.
I very much appreciate your help.

[dfandrich]

NB you've just published your username and password.

[kitsguru]

remove now TY, and password reset

[dfandrich]

There was a fix for an issue where curl would erroneously return "Out of memory" for a bad URL in yesterday's release, but I don't think that's relevant in this case, so the "Out of memory" error may very well be legitimate. Have you tried monitoring memory use of the process when this occurs to see if it's getting very large? The "curl: (28) FTP response timeout" error simply looks like the server is bad or gone down or the connection has gone down.

[kitsguru]

I agree the FTP timeout does not appear to be an issue as it does reconnect.

I opened the activity monitor and when curl first starts, it is using 2.0MB of memory
When the (27) out of memory error occurs it is at 2.3MB

[dfandrich]

It sounds like it's unlikely to be an actual memory issue, but it's still possible. The OOM error occurs after an attempt at allocating memory has failed, so it might be trying to allocate a massive memory buffer for some reason. Does the process have an artificially low max memory set, with ulimit or cgroups or something? Are you able to instrument curl to find out exactly where this error 27 is being set before it exits?

[kitsguru]

Currently:
ulimit -u 2048
ulimit -n 524288
cgroups is not install on my machine.

I don't know what you mean by 'instrument curl' or how to do that.

I have been using iterm2 so tried using terminal and got exactly the same results.

[dfandrich]

2048 means 2 MiB, and you said that when curl return OOM it's using just over 2 MB. That is probably not a coincidence.

[kitsguru]

ulimit -u is the number of processes available and not memory. I am well below that in running processes.
Memory available is unlimited (iterm is using 160MiB)

[dfandrich]

Since git-ftp seems to call a separate curl executable, you should be able to replace it with a special one that logs the location of all the OOM returns so you can see where this OOM path is hit and see if it's a legitimate OOM or not. This might be as simple as compiling a debug version of curl and setting the CURL_MEMDEBUG environment variable when starting git-ftp to a file name to hold the memory trace file that debug curl creates. This might not work if git-ftp forks many curl processes in parallel. Alternately, you could replace the curl executable with a script that starts curl under gdb with breakpoints on all the OOM return lines (I count 627 of these), or using ltrace on malloc/calloc/strdup with backtrace enabled to see which one failed.