setsockopt(SO_SNDBUF, 32768): Not enough space

[eku]

Describe the bug
I'm using ftpd classic on a New 3DS to backup the SD card, which is filled with approx. 50GB. After a while ftp keeps failing with setsockopt(SO_SNDBUF, 32768): Not enough space.

Either used memory is not freed or sockets are not closed.

To Reproduce
Steps to reproduce the behavior:

1. Start ftpdon the 3DS side.
2. Start wget --mirror -c --timout=1 -P <localdir> ftp://<ip_of_3ds>:5000/
3. Wait till wget fails
4. See error output of ftpd on 3DS

Expected behavior
ftpd frees / closes all used resources and transfer does not fail.

Screenshots
N/A

Platform (please complete the following information):

• System N3DS
• System version 11.13.0-45E
• Launch method Homebrew Launcher (Rosaline 2.4.2)
• Type 3dsx
• Classic? Yes 3.1.0

Additional context

Since wget opens a new connection for each transfer, it depends on the number of files to be transferred whether ftpd runs out of resources or not.

[eku]

socket.cpp disables SO_LINGER on 3DS.

Woud you please explain the reason?

Does it mean the socket will never or always stay the 2 minutes in state TIME_WAIT?

BTW slowing down wget does not help. It's still running out of resources.

[mtheall]

setsockopt(SO_SNDBUF, 32768): Not enough space
This appears to be informational only. There is no logic that changes when this fails.

[eku]

setsockopt(SO_SNDBUF, 32768): Not enough space
This appears to be informational only. There is no logic that changes when this fails.

Yes, but the root cause is a resource leak in ftpd. And this needs to be found and closed.

[mtheall]

How long is a while? I've been transferring for hours and not seen this problem.

[mtheall]

Let me know if this works better
ftpd-3dsx.zip

[eku]

How long is a while? I've been transferring for hours and not seen this problem.

Well, it depends on your ftp client. Your client might use one connection to download the entire content of the sd card. wget creates a new connection for every file.

That's why I provided a command line to reproduce the error.

Let me know if this works better

Will do...

[mtheall]

Well, it depends on your ftp client. Your client might use one connection to download the entire content of the sd card. wget creates a new connection for every file.

I was using your command both as-is and also with fixing -timout 1 to --timeout 1

Every client makes a new connection for every file. That's just how FTP works.

[eku]

Both classic and normal ftpd still have the issue.

However, I believe I can narrow down the scenario under which the error occurs.

A fresh mirror does not show the error. However, if I update an existing mirror and a larger file was only partially transferred in a previous run, there is a high probability that the error will occur.

What exactly is happening? wget first tries to download the remaining part via REST. If an error occurs, it switches to RETR. However, if an error occurs with REST, and this happens often, then the memory error occurs immediately after.

Could it be that in this case ftpd does not release resources?

[mtheall]

I will try to reproduce using this new information.

[mtheall]

I found and fixed a bug with REST (it apparently never worked before) so now it should actually resume transfers.

The only way I was able to reproduce your problem was by disconnecting the network cable from my pc during transfer. Seems like 3DS sockets never report errors to poll() (or at least they take longer than I'm willing to wait), so I added an inactivity timeout.

If a session has no socket transfers for 60 seconds, then I end that session, which looks to cleanup this error case. Can you try it out for me?
ftpd-3dsx.zip

After the session was cleaned up, I reconnected my network cable. Then wget errored and retried, and the sync successfully completed.

[eku]

Thank you for your efforts, but unfortunately the error is unchanged. I checked twice to use the correct ftpd.

Log

Größen stimmen nicht überein (lokal 3691428688) -- erneuter Download.

--2024-01-21 09:35:50-- ftp://dizzy:5000/Nintendo%203DS/ed36547026f3bf059688a7a491a11dd4/261100e3471222cf534136340002544d/title/00040000/0f700200/content/00000002.app
=> »/backup/3ds/SDCARD/Nintendo 3DS/ed36547026f3bf059688a7a491a11dd4/261100e3471222cf534136340002544d/title/00040000/0f700200/content/00000002.app«
==> CWD nicht erforderlich.
==> PASV ... fertig. ==> REST 3691428688 ... fertig.
==> RETR 00000002.app ...
Fehler in der Antwort des Servers; Kontroll-Verbindung wird geschlossen.
Erneuter Versuch.

wget retries the REST four more times and then the error occurs on the 3DS.

Instead of pulling the cable, you could truncate an already downloaded file and keep the timestamp (important!). The size of the file also seems to have an influence. Smaller files do not trigger the error.

Would a recording of the network traffic be helpful? Or could you provide an ftpd with debug?

[L4R5]

I tried using curlftpfs for creating incremental backups. This leads to the same issue. From what I can tell from ftpd's log message, curlftpfs is also using the REST command a lot. I can reproduce the error within seconds via downloading a 100 MB file.