Proposal to implement support for ProxyCommand

[vo-va]

There are cases where access to instances through SSH is controlled by IAM policies (as in AWS), and it's mandatory to use tools like this https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html (AWS) or, for GCP, this https://cloud.google.com/iap/docs/using-tcp-forwarding. Both tools support the SSH ProxyCommand option , which simplifies authentication, as seen in this example for AWS https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html .

Currently, it is possible to do this manually i.e., start "forwarding" and then configure Spot to use the tunnel.

I would like to request the implementation of support for ProxyCommand https://man.openbsd.org/ssh_config#ProxyCommand so that starting the tunnel and configuring Spot to use it could be automated. I searched for the keyword "proxycommand" in the code, and it seems that it is not yet implemented. If I missed it, please point me to where I can configure it.

[umputun]

I'm not familiar with proxy forwarding and ProxyCommand; however, we already support agent forwarding. I have no clue if it helps, but just in case you missed it, maybe it can be done this way?

[vo-va]

The forwarding of an SSH agent is a bit different thing.

ProxyCommand is simply an automation to connect to a server using a "Bastion server." For instance, when you need to connect to a Bastion/Jump server first and then from there connect to your private server, you can use ProxyCommand parameter to specify the command that connects you to the Bastion server. The SSH client will execute that command automatically.

AWS and GCP have decided to implement the "Bastion server as a service," example: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html Attached IAM roles and policies to it and other things. This allows more flexible or tighter control over who can connect to the servers. For example, here is an example from AWS on how to use it https://aws.amazon.com/blogs/mt/vr-beneficios-session-manager/

A simple case for AWS would be the following: aws ssm start-session --target “i-XXXXXXXXXXXX”. This command checks your permissions against AWS IAM, and if your permissions are valid, it logs you into the instance and provides an interactive shell. But this tool can be used to create an SSH tunnel. For example, you can specify something like this in your ~/.ssh/config file:

host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
    User ec2-user
    IdentityFile ~/.ssh/aws_key

you will be able to use the SSH client or SCP, and your attempt to access the instance will be checked against IAM permissions.

So technically, using Spot with aws ssm start-session is possible. First, you need to start port forwarding to authenticate yourself on the AWS side.

aws ssm start-session
--target instance-id
--document-name AWS-StartPortForwardingSession
--parameters '{"portNumber":["22"], "localPortNumber":["2222"]}'

Then, configure Spot to use localhost and port 2222 to execute commands on the instance.

I was hoping this could be automated on Spot's side like standard ssh client using it, or at least to know would it be ok to implement, or is it out of scope for Spot's goals?

[umputun]

, or at least to know would it be ok to implement, or is it out of scope for Spot's goals?

I think this shouldn't be too hard to do, and if you want to implement it without inflating the code base significantly, that's fine with me.

[vo-va]

Hi, could you please check this? vo-va@ec8d7b7 This is a work in progress (WIP) and the first iteration. I mainly want to know if the selected approach is somewhat acceptable. If not, could you let me know what should be changed to make it acceptable? I am aware that there are no tests or other necessary elements included yet, as the approach might still need to be adjusted. But I've checked it, and in my case, it works. Here is the debug output.

spot latest
2024/11/25 23:59:04.917 [DEBUG] request to load playbook "spot.yml"
2024/11/25 23:59:04.920 [INFO]  playbook loaded with 7 tasks
2024/11/25 23:59:04.920 [DEBUG] load command "preapare nginx deploy files" (task: pack-nginx)
2024/11/25 23:59:04.920 [DEBUG] load command "aa" (task: testproxy)
2024/11/25 23:59:04.920 [DEBUG] load command "preapare nginx deploy files" (task: deploy-nginx)
2024/11/25 23:59:04.920 [DEBUG] load command "create temp directory on remote host" (task: deploy-nginx)
2024/11/25 23:59:04.920 [DEBUG] load command "deploy nginx configuration" (task: deploy-nginx)
2024/11/25 23:59:04.920 [DEBUG] load command "deploy nginx" (task: deploy-nginx)
2024/11/25 23:59:04.920 [DEBUG] load command "pack" (task: deploy-api-part)
2024/11/25 23:59:04.920 [DEBUG] load command "deliver" (task: deploy-api-part)
2024/11/25 23:59:04.920 [DEBUG] load command "deploy" (task: deploy-api-part)
2024/11/25 23:59:04.920 [DEBUG] load command "create-service" (task: create-systemd-service)
2024/11/25 23:59:04.920 [DEBUG] load command "pack" (task: deploy-cloud-sql-proxy)
2024/11/25 23:59:04.920 [DEBUG] load command "deliver" (task: deploy-cloud-sql-proxy)
2024/11/25 23:59:04.920 [DEBUG] load command "create-service" (task: deploy-cloud-sql-proxy)
2024/11/25 23:59:04.920 [DEBUG] load command "pack" (task: deploy-dashboard)
2024/11/25 23:59:04.920 [DEBUG] load command "deliver" (task: deploy-dashboard)
2024/11/25 23:59:04.920 [DEBUG] load command "deploy" (task: deploy-dashboard)
2024/11/25 23:59:04.920 [INFO]  no inventory loaded
2024/11/25 23:59:04.920 [INFO]  ssh key: /Users/vovadev/.ssh/google_compute_engine
2024/11/25 23:59:04.920 [DEBUG] use private key "/Users/vovadev/.ssh/google_compute_engine"
2024/11/25 23:59:04.920 [DEBUG] use ssh agent
2024/11/25 23:59:04.920 [DEBUG] runner created: concurrency:1, connector: ssh connector with private key /Users/v.., timeout 30s, agent true, ssh_shell:"", verbose:true, dry:false, only:[], skip:[]
2024/11/25 23:59:04.920 [DEBUG] task "testproxy" has 1 commands
2024/11/25 23:59:04.920 [DEBUG] target "dev" found in playbook
2024/11/25 23:59:04.920 [DEBUG] target "dev" has 1 hosts: [{Name:gc-tf Host:gc-tf Port:22 User:vova Tags:[] ProxyCommand:gcloud compute start-iap-tunnel %h 22 --listen-on-stdin --project=tutorial-777 --zone=europe-central2-a --verbosity=warning}]
2024/11/25 23:59:04.920 [DEBUG] target "dev" has 1 total hosts: [{Name:gc-tf Host:gc-tf Port:22 User:vova Tags:[] ProxyCommand:gcloud compute start-iap-tunnel %h 22 --listen-on-stdin --project=tutorial-777 --zone=europe-central2-a --verbosity=warning}]
2024/11/25 23:59:04.920 [DEBUG] target hosts (1) [{Name:gc-tf Host:gc-tf Port:22 User:vova Tags:[] ProxyCommand:gcloud compute start-iap-tunnel %h 22 --listen-on-stdin --project=tutorial-777 --zone=europe-central2-a --verbosity=warning}]
2024/11/25 23:59:04.920 [DEBUG] connect to "gc-tf:22" (gc-tf), user "vova"%!(EXTRA string=gcloud compute start-iap-tunnel %h 22 --listen-on-stdin --project=tutorial-777 --zone=europe-central2-a --verbosity=warning)
2024/11/25 23:59:04.920 [DEBUG] create ssh session to gc-tf:22, user vova
2024/11/25 23:59:04.920 [DEBUG] ProxyCommand gcloud compute start-iap-tunnel %h 22 --listen-on-stdin --project=tutorial-777 --zone=europe-central2-a --verbosity=warning
2024/11/25 23:59:04.920 [DEBUG] ssh agent found at /private/tmp/com.apple.launchd.777/Listeners
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

2024/11/25 23:59:06.394 [DEBUG] ssh session created to gc-tf:22
[gc-tf:22] run task "testproxy", commands: 1
2024/11/25 23:59:06.394 [INFO]  run command "aa" on host "gc-tf:22" (gc-tf)
[gc-tf:22] run command "aa"
2024/11/25 23:59:06.394 [DEBUG] execute script "aa" on gc-tf:22
2024/11/25 23:59:06.395 [DEBUG] command "aa" is multiline, using script file
2024/11/25 23:59:06.396 [DEBUG] upload /var/folders/tv/l0l32mdx14zgwrm2ssn1hmwh0000gp/T/spot-script1225299126 to /tmp/.spot-943147986530432768/spot-script1225299126
2024/11/25 23:59:06.396 [DEBUG] upload /var/folders/tv/l0l32mdx14zgwrm2ssn1hmwh0000gp/T/spot-script1225299126 to gc-tf:/tmp/.spot-943147986530432768/spot-script1225299126
2024/11/25 23:59:06.628 [DEBUG] file mode for /var/folders/tv/l0l32mdx14zgwrm2ssn1hmwh0000gp/T/spot-script1225299126: 0700
2024/11/25 23:59:06.712 [INFO]  uploaded /var/folders/tv/l0l32mdx14zgwrm2ssn1hmwh0000gp/T/spot-script1225299126 to gc-tf:/tmp/.spot-943147986530432768/spot-script1225299126 in 316.047083ms
2024/11/25 23:59:06.712 [DEBUG] run /bin/sh -c /tmp/.spot-943147986530432768/spot-script1225299126
2024/11/25 23:59:06.712 [DEBUG] run ssh command "/bin/sh -c /tmp/.spot-943147986530432768/spot-script1225299126" on pipe
[gc-tf:22]  > /bin/sh -c /tmp/.spot-943147986530432768/spot-script1225299126
[gc-tf:22]  > Prepare nginx configs
[gc-tf:22]  > total 24
[gc-tf:22]  > drwxr-xr-x 3 vova vova 4096 Nov 25 22:40 .
[gc-tf:22]  > drwxr-xr-x 5 root             root             4096 Nov 25 22:41 ..
[gc-tf:22]  > -rw-r--r-- 1 vova vova  220 Nov 25 22:40 .bash_logout
[gc-tf:22]  > -rw-r--r-- 1 vova vova 3771 Nov 25 22:40 .bashrc
[gc-tf:22]  > drwx------ 2 vova vova 4096 Nov 25 22:40 .cache
[gc-tf:22]  > -rw-r--r-- 1 vova vova  807 Nov 25 22:40 .profile
2024/11/25 23:59:06.867 [INFO]  deleted recursively /tmp/.spot-943147986530432768
[gc-tf:22] completed command "aa" {script: /bin/sh -c [multiline script]} (482ms)
[gc-tf:22] completed task "testproxy", commands: 1 (1.957s)
2024/11/25 23:59:06.878 [INFO]  completed: hosts:1, commands:1 in 1.9s
2024/11/25 23:59:06.878 [INFO]  completed all 1 targets in 1.9s