Node provisioning stuck on Waiting for ssh when provisioning from an airgapped environment with proxy

[bmdepesa]

rancher/rancher:v2.4.5

• Create an airgapped single-node rancher setup which is configured to use a proxy for outbound connections
  • https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/air-gap/
  • https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/proxy/
• Create a nodedriver cluster (ec2,do)
• The node is created in the cloud provider, but provisioning hangs on Waiting for SSH to be available

Expected:

• The node provisions successfully

Notes:

• The Rancher instance is accessible through a public NLB to external machines
• Outbound connections from Rancher are configured to go through the proxy
• curl commands from within the Rancher container go through the proxy successfully
• ssh from within the Rancher container fails unless using a jumphost
  • exec into the Rancher container
  • ssh to some machine (in this case, some random machine on ec2 in a different VPC)
  • https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts

root@35d3c0b70171:/var/lib/rancher# ssh -i mykey.pem -o ProxyCommand="ssh -W %h:%p host.compute.amazonaws.com:3131" ubuntu@host
root@host.compute.amazonaws.com: Permission denied (publickey).
ssh_exchange_identification: Connection closed by remote host

# this example above fails due to the key error, but typically this would time out 

root@35d3c0b70171:/var/lib/rancher# ssh ubuntu@host
ssh: connect to host <host> port 22: Connection timed out

• The same configuration in a non-airgapped setup (but still using a proxy) allows a cluster to be provisioned successfully, though no ssh traffic is seen traversing the proxy

[Jono-SUSE-Rancher]

Does this now work with the proxy changes that were made in 2.5.6?

[bmdepesa]

@slickwarren can you retest this?

[slickwarren]

Rancher/rancher:v2.5.6 I was able to reproduce this issue:

• Create an airgapped node, then configure it to use a proxy for outbound connections. Deploy rancher via single-install setup instructions
  • https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/air-gap/
  • https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/proxy/
• Create a nodedriver cluster (ec2,do)
• The node is created in the cloud provider, but provisioning hangs on Waiting for SSH to be available

Expected:

• The node provisions successfully

Notes:

• The Rancher instance is accessible through a public NLB to external machines
• Outbound connections from Rancher are configured to go through the proxy
• curl commands from within the Rancher container go through the proxy successfully
• ssh from within the Rancher container fails when using public IP of other nodes
  • exec into the Rancher container
  • ssh to some machine (in this case, some random machine on ec2 in a different VPC)
  • https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts

root@35d3c0b70171:/var/lib/rancher# ssh ubuntu@host
ssh: connect to host <host> port 22: Connection timed out

• The same configuration in a non-airgapped setup (but still using a proxy) allows a cluster to be provisioned successfully, though no ssh traffic is seen traversing the proxy

[jiaqiluo]

Root cause

Rancher invokes the embedded rancher-machine binary to provision nodes in the cluster providers, but the rancher-machine lacks the ability to use proxy to connect to the cluster providers.

What was fixed, or what changes have occurred

• in rancher-machine, add the support for using the proxy if proxy-related env vars are detected
• in Rancher, update the rancher-machine binary to the new version, and package new dependencies into the image
• in Rancher, update the command to let rancher-machine use external ssh binary which can use proxy

Areas or cases that should be tested

• provision node-driver clusters when Rancher is in the airgap environment and behind a proxy

What areas could experience regressions?

none, this is a new feature

Are the repro steps accurate/minimal?

yes

[slickwarren]

I'm still seeing this issue on v2.6-head (e7f03fd):

reproduction steps:

• bring up a proxy-gapped rancher server
• provision an rke1 cluster (digital ocean, all default settings)
  results in this error:

Notifying bugsnag: [Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded]

I am able to provision an rke2 cluster with default settings in the same setup.

[jiaqiluo]

The failure is caused by the misconfiguration of the proxy server.
moved it back to to-test.

[slickwarren]

after adding the following to the proxy config, I was able to continue testing

acl SSL_ports port 22
acl SSL_ports port 2376

acl Safe_ports port 22      # ssh
acl Safe_ports port 2376    # docker port 

on v2.6-head (e7f03fd) using a proxy-gapped rancher server with the above configuration of the proxy:

• provision an rke1 cluster (digital ocean, single node all roles) -- pass
• provision rke2 cluster (digital ocean, single node all roles) -- pass
• provision rke1 cluster with dedicated roles (aws) -- pass
  • deploy and test workloads and services -- pass
• provision rke2 cluster with dedicated roles (aws) -- pass
  • deploy and test workloads and services -- pass