Windows release builds failing with `subsystem request failed on channel 0`

[richardlau]

This is blocking the 18.4.0 release.

e.g. https://ci-release.nodejs.org/job/iojs+release/nodes=vs2019-x64/8531/console

02:29:04 Successfully signed: node-v18.4.0-x64.msi

02:29:04 Successfully signed node-v18.4.0-x64.msi using timeserver http://timestamp.comodoca.com/authenticode
02:29:05 subsystem request failed on channel 0
02:29:05 scp: Connection closed
02:29:05 Build step 'Conditional steps (multiple)' marked build as failure

It looks like the signing step has completed. The next thing vcbuild.bat is supposed to do is upload the artifacts to staging: https://github.com/nodejs/node/blob/f4c830fbe49b0235c6985a736a40a650a9be7530/vcbuild.bat#L540-L570

[richardlau]

Builds started failing yesterday (14 June) on both release machines for both x64 and x86:

• release-rackspace-win2012r2_vs2019-x64-1
• release-rackspace-win2012r2_vs2019-x64-2

[richardlau]

It looks like release-rackspace-win2012r2_vs2019-x64-1 is able to ssh to staging but scp commands from it to copy to staging are failing 😕.

[richardlau]

This is from the logs on the staging server:

Jun 15 06:26:59 infra-digitalocean-ubuntu1604-x64-1 sshd[21218]: pam_unix(sshd:session): session opened for user staging by (uid=0)
Jun 15 06:26:59 infra-digitalocean-ubuntu1604-x64-1 systemd: pam_unix(systemd-user:session): session opened for user staging by (uid=0)
Jun 15 06:26:59 infra-digitalocean-ubuntu1604-x64-1 systemd-logind[16867]: New session 306559 of user staging.
Jun 15 06:26:59 infra-digitalocean-ubuntu1604-x64-1 sshd[21260]: subsystem request for sftp by user staging failed, subsystem not found
Jun 15 06:26:59 infra-digitalocean-ubuntu1604-x64-1 sshd[21218]: pam_unix(sshd:session): session closed for user staging
Jun 15 06:26:59 infra-digitalocean-ubuntu1604-x64-1 systemd: pam_unix(systemd-user:session): session closed for user staging

[richardlau]

Possibly related is that it looks like git was updated on the machines between the last successful build and the failing ones.

• https://ci-release.nodejs.org/job/windows-update-reboot/label=release-rackspace-win2012r2_vs2019-x64-1/1839/console
• https://ci-release.nodejs.org/job/windows-update-reboot/label=release-rackspace-win2012r2_vs2019-x64-2/1839/console

Details

You have git.install v2.36.0 installed. Version 2.36.1 is available based on your source(s).

Progress: Downloading git.install 2.36.1... 0%
...
Progress: Downloading git.install 2.36.1... 100%

git.install v2.36.1 [Approved]
git.install package files upgrade completed. Performing other installation steps.
Using Git LFS
Installing 64-bit git.install...
git.install has been installed.
WARNING: No registry key found based on  'Git version *'
WARNING: Can't find git.install install location
  git.install can be automatically uninstalled.
 The upgrade of git.install was successful.
  Software installed to 'C:\Program Files\Git\'
...
Chocolatey upgraded 1/32 packages. 
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Upgraded:
 - git.install v2.36.1

[rvagg]

You're right on the culprit,

git for Windows 2.36.1: https://github.com/git-for-windows/git/releases/tag/v2.36.1.windows.1

Comes with OpenSSH v9.0p1.

Which says:

This release switches scp(1) from using the legacy scp/rcp protocol to using the SFTP protocol by default.

And

In case of incompatibility, the scp(1) client may be instructed to use the legacy scp/rcp using the -O flag.

But I don't think that's an easy option here because it's in the build scripts.

I think we're just going to have to install sftp on the server and be done with it. This is probably fairly safe, it nests within sshd as a subsystem rather than opening a new port I believe. @richardlau if you agree, then could you apt-get install openssh-sftp-server on the server, see if it's enabled in /etc/ssh/sshd_config and then try the builds again to see if they work?

[richardlau]

Thanks @rvagg. It looks like we already had openssh-sftp-server installed but not enabled. I've added

Subsystem sftp internal-sftp
  Match Group staging
  ForceCommand internal-sftp

to /etc/ssh/sshd_config, restarted sshd and verified I can sftp in from one of the release machines.

Test build: https://ci-release.nodejs.org/job/iojs+release/8534/

[richardlau]

I've got something wrong with the setup:

This service allows sftp connections only.

[richardlau]

Removed the ForceCommand line and trying again: https://ci-release.nodejs.org/job/iojs+release/8535/

[richardlau]

Looks fixed now 🎉.