scp starts to throw "too many files" with large recursive transfer

[cbookg]

Please answer the following
"OpenSSH for Windows" version

• v7.6.1.0p1-Beta
• also reproduced after building scp from head.

Server OperatingSystem
Linux - Ubuntu

Client OperatingSystem
Windows 10 Enterprise

What is failing
Steps:

• Run "scp -r * targethost:/target/path" in a directory structure containing 8500 files.

Expected output

• All files transfered properly.

Actual output

• The last 50 files fail to transfer. The output looks something like this:
  ...
  filethatworks1 100% 6192 6.1KB/s 00:00
  filethatworks2 100% 894 0.9KB/s 00:00
  filethatfails1: eother
  filethatfails2: eother
  ...

I reproduced this by building from head and setting breakpoints to figure out more details about the failure.
The errors are coming from the win32 specific fileio_fstat() function. The call to _open_osfhandle returns 24, which appears to mean "too many open files".

It appears as though scp is leaking file handles or something. Is this a known issue?

[bagajjal]

As per MSDN, _open_osfhandle returns -1 in case of failure. You mentioned it returned 24 which means it's successful and failing some where else?

[cbookg]

Sorry, it does return -1. The value of errno when it does is 24.

[cbookg]

The problem seems to be that this function is using _open_osfhandle which creates a new "runtime file descriptor", stored in fd but never closed (so it is leaked). We can't close this directly as the windows handle is still used outside of this function. ie from the docs "The _open_osfhandle call transfers ownership of the Win32 file handle to the file descriptor. To close a file opened with _open_osfhandle, call _close"

Instead, it looks like we should duplicate the windows handle, and then ensure we close the fd created from _open_osfhandle after the _fstat64:

fileio_fstat(struct w32_io* pio, struct _stat64 *buf)
{
int fd = _open_osfhandle((intptr_t)pio->handle, 0);
debug4("fstat - pio:%p", pio);
if (fd == -1) {
errno = EOTHER;
return -1;
}

return _fstat64(fd, buf);

}

[NoMoreFood]

@bagajjal His change looks good to me.

[manojampalam]

@cbookg thanks for following up. I'll pull in your fix.