Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SOLVED] put hangs on bigger files to certain hostings #342

Closed
ljacho opened this issue Aug 15, 2021 · 11 comments
Closed

[SOLVED] put hangs on bigger files to certain hostings #342

ljacho opened this issue Aug 15, 2021 · 11 comments

Comments

@ljacho
Copy link

ljacho commented Aug 15, 2021

If I am trying to upload small file colors.js everything works like a breeze. As soon as I want to do the same with main.js. Put hangs.

-rw-rw-r-- 1 sbtge sbtge   167 Aug 15 22:42 colors.js
-rw-rw-rw- 1 sbtge sbtge 11833 Aug 15 22:49 main.js

Here is the log.

[2021-08-15T22:49:10.698] INFO:   🔼 [SFTPClient] uploading main.js
[2021-08-15T22:49:10.698] CLIENT[sftp]: put: Adding temp event listeners
[2021-08-15T22:49:10.699] Outbound: Sending CHANNEL_DATA (r:0, 63)
[2021-08-15T22:49:10.699] SFTP: Outbound: Buffered OPEN
[2021-08-15T22:49:10.699] CLIENT[sftp]: put source is a stream
[2021-08-15T22:49:10.731] Inbound: CHANNEL_DATA (r:0, 17)
[2021-08-15T22:49:10.732] SFTP: Inbound: Received HANDLE (id:0)
[2021-08-15T22:49:10.732] Outbound: Sending CHANNEL_DATA (r:0, 25)
[2021-08-15T22:49:10.732] SFTP: Outbound: Buffered FSETSTAT
[2021-08-15T22:49:10.765] Inbound: CHANNEL_DATA (r:0, 28)
[2021-08-15T22:49:10.765] SFTP: Inbound: Received STATUS (id:1, 0, "Success")
[2021-08-15T22:49:10.766] Outbound: Sending CHANNEL_DATA (r:0, 7320)
[2021-08-15T22:49:10.766] SFTP: Outbound: Sent WRITE (id:2)
[2021-08-15T22:49:12.645] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:14.645] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:16.645] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:18.646] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:20.647] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:22.647] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:24.647] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:26.648] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:28.648] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:30.648] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:32.649] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:34.649] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:36.649] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:38.650] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:40.650] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:42.650] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:44.650] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:46.650] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:48.650] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:50.651] Outbound: Sending ping (GLOBAL_REQUEST: keepalive@openssh.com)
[2021-08-15T22:49:52.652] CLIENT[sftp]: put: Handling error: Keepalive timeout
[2021-08-15T22:49:52.652] CLIENT[sftp]: put: handled error with reject
[2021-08-15T22:49:52.652] CLIENT[sftp]: Global: Ignoring handled error
[2021-08-15T22:49:52.652] CLIENT[sftp]: put: Removing temp event listeners
[2021-08-15T22:49:52.653] Error: put: Keepalive timeout
    at fmtError (/home/sbtge/workspace/1-lab/ftp-syncer/dist/index.js:3767:22)
    at Client.fn (/home/sbtge/workspace/1-lab/ftp-syncer/dist/index.js:3782:20)
    at Client.emit (node:events:388:22)
    at Timeout.sendKA (/home/sbtge/workspace/1-lab/ftp-syncer/node_modules/ssh2/lib/client.js:662:16)
    at listOnTimeout (node:internal/timers:556:17)
    at processTimers (node:internal/timers:499:7)

I am using ssh2-sftp-cllient: 7.0.1 and node v. 15.3.0.
@theophilusx
Copy link
Owner

theophilusx commented Aug 15, 2021 via email

@ljacho
Copy link
Author

ljacho commented Aug 16, 2021 •
edited

Thank you for your quick answer. I changed the node to 16.6.2 as you advised. Problem persists.

Here is a simplified version of the code:

index.ts :

const sFTP = new SFTP();
sFTP.pub('/home/myUser/test/main.js');

sftp.ts:

import Client from 'ssh2-sftp-client';

export default class SFTP {
  constructor() {
     this.options = {
      host: config.sftp.host,
      port: config.sftp.port,
      username: config.sftp.username,
      password: config.sftp.password,
      debug: config.sftp.debug ? console.log : null,
    }
   this.uploadOptions = {
       flags: 'w', // w - write and a - append
      encoding: null, // use null for binary files
      mode: 0o664, // mode to use for created file (rwx)
      autoClose: true, // automatically close the write stream when finished
   }
  }

  public pub(path: string): void {
      const data = fs.createReadStream(path);
      this.client = new Client();
  
      const fileName = path.split('/').pop();
  
      this.client
        .connect(this.options)
        .then(() => {
          console.log('uploading');
          return this.client.put(data, `${config.sftp.remotePath}/${fileName}`, this.uploadOptions);
        })
        .then(() => {
          console.log('uploaded! ending');
          return this.client.end();
        })
        .catch((err) => {
          console.log(`Error: ${err.message}`); // error message will include 'example-client'
        });
    }
}

Code is heavily inspired by your examples. Thank you for any help!

@ljacho
Copy link
Author

ljacho commented Aug 16, 2021 •
edited

UPDATE: I found out that files ~3.06 KiB and bigger are not uploading correctly. If I use a client e.g. Filezilla, all files are being uploaded without problems.

@theophilusx
Copy link
Owner

theophilusx commented Aug 16, 2021 via email

@ljacho
Copy link
Author

ljacho commented Aug 16, 2021 •
edited

options besides password/username/host I got port 22

and

   uploadOptions = {
       flags: 'w', // w - write and a - append
      encoding: null, // use null for binary files
      mode: 0o664, // mode to use for created file (rwx)
      autoClose: true, // automatically close the write stream when finished
   }

sftp server is running on godaddy - I dont know what platform are they using.

IMPORTANT: I found out that it is not working in WSL2 but on mac x64 it works with bigger files!

Do you have an idea where could be the problem?

@theophilusx
Copy link
Owner

theophilusx commented Aug 16, 2021 via email

@ljacho
Copy link
Author

ljacho commented Aug 16, 2021

thank you for deeper explanation! I will try to continue with this for a bit ;)

@theophilusx
Copy link
Owner

theophilusx commented Aug 16, 2021 via email

@ljacho
Copy link
Author

ljacho commented Aug 16, 2021 •
edited

gotcha!

look:

I can upload main.js (that bigger file) to another sftp server running on digital ocean vps without any problems

  • from mac x64
  • from wsl2

I can upload main.js to godaddy

  • from mac x64
  • from wsl2 - Ubuntu
  • UPDATE: Ubuntu 20.04.1 LTS

even linux sftp CLI utility hangs with uploading main.js from WSL2 - Ubuntu & Ubuntu 20.04.1 LTS to godaddy!

we can close this as it's not the problem of ssh2-sftp-client at all

@ljacho ljacho changed the title put hangs on bigger files [SOLVED] put hangs on bigger files -> uploading ~3.5KiB files from WSL2 to certain hostings Aug 16, 2021
@ljacho ljacho changed the title [SOLVED] put hangs on bigger files -> uploading ~3.5KiB files from WSL2 to certain hostings [SOLVED] put hangs on bigger files -> uploading ~3.5KiB files to certain hostings Aug 16, 2021
@ljacho
Copy link
Author

ljacho commented Aug 17, 2021

definitely not a problem of ssh2-sftp-client

I prepared answer for all people coming here.

Short Answer
You probably have an MTU/fragmentation problem. For each network interface on both client and server set the MTU to 576, eg ifconfig eth0 mtu 576. If the problem goes away, read on.

Long Answer
Long answer: At each routing hop, IP packets bigger than the outgoing interface's Maximum Transmission Unit (MTU) get fragmented. Only the first fragment has TCP port numbers. Firewalls often behave badly in the presence of packet fragmentation, dropping everything but the first fragment since the subsequent ones can't be matched against the firewall rules. Some NAT configuration (eg many-to-one NAT or port address translation) can't match the fragments against their translation state tables.

Arguably, such devices should perform packet reassembly first so as to properly consider fragmented packets. However, this is more complicated and so is often not done. Also, this feature would raise a possible starvation attack against the packet filter, by sending many bogus initial fragments and causing the device to store them for reassembly with subsequent packets which will never come.

Logging in and using the shell will normally generate relatively small packets, and so the initial connection proceeds normally ; however if do you something that generates a lot of data (eg cat'ing a big file or starting an X Windows application), you may generate a packet bigger than the MTU.

Let's say it's a 1500 byte IP packet and the router has 2 different MTU's (say 1500 & 1484) and no firewall. When the router goes to forward it, the packet is too big for the interface MTU (1484), so the router breaks it into 2 fragments, 0 and 1. Fragment 0 contains the first 1484 bytes (including the TCP source and dest ports) and fragment 1 contains the remaining 16 bytes. Both fragments are sent on to their destinations.

When the first fragment reaches its target, it's held by the IP stack until the remaining fragments arrive, at which time the IP packet is reassembled and passed up the stack to TCP. If all fragments are not received by the timeout, the entire IP packet is discarded and an ICMP "timeout during reassembly" error is sent back.

Now add your firewall, which drops fragment 1. Your 1500 byte IP packet times out during reassembly and TCP retries, by sending another 1500 byte packet. Repeat. Eventually, TCP will time out and you'll get a connection termination.

IP stack parameters (such as Path MTU Discovery) and external variables (such as the MTU's of all the hops between hosts) can also affect whether or not a given connection will have this problem.

source: snailbook.com

temporary solution:

ip link set eth0 mtu 1400

@ljacho ljacho changed the title [SOLVED] put hangs on bigger files -> uploading ~3.5KiB files to certain hostings [SOLVED] put hangs on bigger files to certain hostings Aug 17, 2021
@theophilusx
Copy link
Owner

theophilusx commented Aug 18, 2021 via email •
edited

@tristansokol tristansokol mentioned this issue Apr 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@theophilusx @ljacho