Collecting logs from EKS Worker Nodes running Bottlerocket AMI when no SSH is enabled

[maiconrocha]

The documentation on the link says that we can use [SSH over SSM] when no SSH is enabled:

bottlerocket/README.md

Line 483 in 1353ea1

(If your instance isn't accessible through SSH, you can use [SSH over SSM](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html).)

however I tried following those steps and this does not seem to work on Bottlerocket
when trying to edit the ssh config file it says:

"~/.ssh/config" E212: Can't open file for writing 

even from inside the admin container.

I was able to find a solution which works, only SSM access is required:

Steps are:

1. SSM to node
2. enter-admin-container
3. sudo sheltie
4. logdog

logs will be generated at: /var/log/support/bottlerocket-logs.tar.gz

then we can run

5. kubectl get --raw "/api/v1/nodes/<node>/proxy/logs/support/bottlerocket-logs.tar.gz" > bottlerocket-logs.tar.gz

and it will transfer the bottlerocket-logs.tar.gz to the local machine.

If Bottlerocket team believes this is an approved solution and the README needs to be updated with above solution feel free to do so or let me know, and I can create a PR.
Thanks.

[koooosh]

Hello, thanks for reaching out and creating the issue. Bottlerocket is purposefully designed not have an SSH server, which is why you cannot edit the SSH config on the Bottlerocket host.

Looking at that SSH over SSM link, Step 2 states:

"On the local machine from which you want to connect to a managed node using SSH, do the following..."

So in this case, you want to edit the ssh config on your local machine. Please add the following to your local ssh config:

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

and then follow the steps in the Logs section.

However, the workaround you've identified is great; it's specific to users of aws-k8s nodes. If you'd like, feel free to PR this section of the README to include your workaround as something like:

Or, if you're running kubernetes, you can retrieve the file with: kubectl get --raw "/api/v1/nodes/<node>/proxy/logs/support/bottlerocket-logs.tar.gz" > bottlerocket-logs.tar.gz