Error: Timeout (control socket)

[nezzard]

Describe the bug
When i upload large file, i have problem with timeout, file upload full, but not resolving and i'm gettings an error Socket Timeout..

Example code
await client.uploadFromDir("videos/" + dirName, "videos/" + dirName).then(() => { console.log('Resolved') resolve() }).catch((err) => { console.log('Rejected') client.close() reject(err) })

Console output
Error: Timeout (control socket)
at TLSSocket. (C:\croper\node_modules\basic-ftp\dist\FtpContext.js:310:58)
at Object.onceWrapper (events.js:308:28)
at TLSSocket.emit (events.js:219:5)
at TLSSocket.Socket._onTimeout (net.js:476:8)
at listOnTimeout (internal/timers.js:536:17)
at processTimers (internal/timers.js:480:7)

Which version of Node.js are you using?
12.13.0

[patrickjuchli]

Could you post the whole log output, not just the exception?

My guess is that this is probably a typical problem of FTP. I will use this issue to describe the problem and document the current policy of this library also for future reference.

An FTP client uses two connections, one to send commands and receive answers (control), and another one for file transfer (data). For file transfer it is actually a completely new connection for each file but this is not relevant here. During a long file transfer over the data connection, the control connection sits idle. It is possible that during this time, a router somewhere between client and server disconnects the control connection because it seems inactive. This is not standard behaviour at all but it can happen. Routers either usually don't do this or then do indeed treat FTP differently by inspecting traffic (!) and know to keep the control connection open. With TLS encryption when using FTPS though, the router can't inspect traffic and doesn't know that control and data connection actually belong to an FTP client session. From the outside, these two connections are unrelated and the idle one may be closed.

My guess is that this is what's happening in your case. Now, what can you do about it?

Some FTP clients send some stuff during file transfer at regular intervals over the idle control connection (NOOP command or something else like PWD). This library opted to not do that because it is not standard behaviour at all and doesn't work with some servers. The technique has other sideeffects. The current policy is: If you transfer large files that take a long time, you have to take into account that there might be a connection problem anyway just because it's the internet. You will have to handle this in your code and resume transfer. By doing that you'll automatically handle the described problem here as well.

Why are automatic retries not built-in? When developing a library like this one, it's important to define how far you want to go. Is this the model layer for a full-fledged FTP client and all you'd have to do is add a user interface? Or is it one level lower – it covers just the basics of FTP but everything else has to be provided by the programmer. This library took the second approach. The target audience is other programmers. Automatic retries of long-running tasks is not an FTP problem.

Now. Looking at how this library has been used over the last years I can see that a lot of developers have a different understanding, especially beginners. They just don't know about these problems and do expect the library to handle everything.

I'll reconsider the decision around this topic. I see that it might be helpful. But it doesn't have a high priority right now.

[patrickjuchli]

May I ask how much time the file transfer took in your case?

[patrickjuchli]

@nezzard, I'll close this issue if there is nothing more from your side.

[foofaev]

Hope, you don't mind if I comment here
I've got the same problem, at least it looks so.

A few words about environment:

• ubuntu 19.10, node 8.17.0, npm 6.14.4 - no error
• ubuntu 19.10, node 13.11.0, npm 6.13.7 - no error
• ubuntu server 18.04, node v8.10.0, npm 3.5.2 - error!

AccessOptions:

• host: 'mx.some-host.org'
• user: some_user
• password: some_passwod

Log:

Connected to 111.111.111.11:21
< 220-FileZilla Server 0.9.60 beta
220-written by Tim Kosse (tim.kosse@filezilla-project.org)
220 Please visit https://filezilla-project.org/
Login security: No encryption
> USER some_user
< 331 Password required for some_user
> PASS ###
< 230 Logged on
> TYPE I
< 200 Type set to I
> STRU F
< 200 Using file structure 'File'
> OPTS UTF8 ON
< 202 UTF8 mode is always enabled. No need to send this command.
> OPTS MLST type;size;modify;unique;unix.mode;unix.owner;unix.group;unix.ownername;unix.groupname;
< 200 MLST OPTS type;size;modify;
Trying to find optimal transfer strategy...
> EPSV
< 229 Entering Extended Passive Mode (|||57996|)
Optimal transfer strategy found.
> MLSD
< 150 Opening data channel for directory listing of "/"
226 Successfully transferred "/"
Downloading from undefined:undefined (No encryption)

After that socket hangs, lib throws timeout error

Error: Timeout (control socket)
at TLSSocket. (C:\croper\node_modules\basic-ftp\dist\FtpContext.js:310:58)
at Object.onceWrapper (events.js:308:28)
at TLSSocket.emit (events.js:219:5)
at TLSSocket.Socket._onTimeout (net.js:476:8)
at listOnTimeout (internal/timers.js:536:17)
at processTimers (internal/timers.js:480:7)

Solution:
use ip address instead of url resolves error

[patrickjuchli]

The last line of the first log shows Downloading from undefined:undefined (No encryption). I remember issues mentioning the same thing (#107) they've been resolved with newer versions of Node. Apparently already 8.17.0 might do the trick.

[psr1410]

Hi Patrick, We also facing the same issue while downloading larger files(2GB to 6 GB we tested). The control_socket timed out issue. All the file stream was downloaded without any problems(It took around 1hr to download 6GB file). But the real problem occurs here while I'm uploading the downloaded stream to Azure storage it is throwing that control_socket TIMEDOUT issue.

I did what you suggested, Checking the client connection closed or not after each stream chunk downloaded. Interestingly the client connection was active, till the total file downloaded.

How can I check and make control_socket connection active? Can you help me out on this issue?