aws ssm start-session end with **Cannot perform start session: EOF**

[alxsbn]

I try to instanciate a vanilla aws ssm start-session on a Github actions Jobs (without SSH).

The session is well instanciated (i can see it on the AWS Session Manager dashboard) but always finish with a Cannot perform start session: EOF

FYI I can instanciated a sesion outside of GH actions (.e.g. on my laptop) with the same account / role / policy / instances (ami-0ea4a063871686f37).

Here's my job YAML :

name: CI

# Controls when the action will run. 
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      - name: Configure AWS credentials from Test account
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
                   
      - name: Initialize a Session Manager session
        shell : sh
        run: aws ssm start-session --target ${{ secrets.TARGET_INSTANCE }}

The results :

Run aws ssm start-session --target ***
  aws ssm start-session --target ***
  shell: /bin/sh -e {0}
  env:
    AWS_DEFAULT_REGION: ***
    AWS_REGION: ***
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***

Starting session with SessionId: gh-ssm-0d50831b64f0xxxxx
$ Cannot perform start session: EOF

Here are the CLI tools packaged with Ubuntu-latest virtual env (20.04.2 LTS) :

• AWS CLI 2.1.24
• AWS CLI Session manager plugin 1.2.54.0

[nitikaaws]

Thanks for reaching out to us!

Could you please share logs for Session Manager plugin for us to investigate this further? More information on logging can be found here - https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-configure-logs.

[keshavstitcher]

Did you get any resolution on this?

I am also facing the same issue when I tried to login via Jenkins job. while using the command line it's working fine.

[olfway]

I had similar issue with "aws ecs execute-command ..." and it seems it's related to tty being not available when running in ci

I fixed it with "unbuffer aws ecs execute-command ..." to run it with tty available

[alxsbn]

It's linked to tty. I bypass the problem with ssm send-command with remote script execution

[emmahsax]

@alxsbn Can you explain more what you meant by "ssm send-command with remote script execution"? We're running into the same issue with

aws ecs execute-command

I don't know if this makes a difference, but we're using ECS with Fargate, not EC2.

[alxsbn]

@emmahsax Our first reflex was to run a ssh command, then the others commands we need (for us it was linked to CD, so rsync a repo then restart a pm2 service). Since we have this tty problem we just write a shell script with our commands an fire him with ssm send-command

[dgholz]

I have the same problem, I followed https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-shell-config.html to add bash -i to my Linux Shell Profile

I enabled logging and see:

2021-05-11 11:10:09 DEBUG Start File Watcher On: /usr/local/sessionmanagerplugin/seelog.xml
2021-05-11 11:10:09 DEBUG Start Watcher on directory: /usr/local/sessionmanagerplugin
2021-05-11 11:10:09 DEBUG Calling Initialize Datachannel for role: publish_subscribe
2021-05-11 11:10:09 INFO Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe)
2021-05-11 11:10:09 INFO Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe)
2021-05-11 11:10:09 INFO Sending token through data channel wss://ssmmessages.eu-west-1.amazonaws.com/v1/data-channel/botocore-session-1620726103-0af510cb5dd7ad804?role=publish_subscribe to acknowledge connection
2021-05-11 11:10:09 DEBUG WebsocketChannel: Send ping. Message.
2021-05-11 11:10:10 DEBUG Processing stream data message of type: output_stream_data

and the logs end there.

I saw the same issue when running aws ssm start-session --target INSTANCE_ID --document-name AWS-StartInteractiveCommand --parameters command="bash -i":

Starting session with SessionId: botocore-session-1620726103-06d6cbb9a7fcd54a6
ssm-user@ip-10-129-212-60:/var/snap/amazon-ssm-agent/3552$ Cannot perform start session: EOF

It also messes up the terminal, I have to reset to get what I type to show properly again after it exits.

I'm using version 1.2.54.0

[nitikaaws]

Session manager plugin has been designed to be invoked from a terminal for shell scenarios so it can interpret control sequences returned by pseudo terminal and display appropriately. When a shell session is started, session manager plugin also fetches terminal size on client machine and sends it to pseudo terminal on target server. We suspect one of these actions might be failing for you. Please let us know if you are invoking session manager plugin from a terminal and still seeing error of Cannot perform start session: EOF.

[dgholz]

Yes, I am invoking it from a terminal. I haven't changed anything about the instances I'm connecting to, they're launched from an AMI I baked weeks ago. It's working today, without changing anything in my local environment (same version of session-manager-plugin, same AMI/instances)

When a shell session is started, session manager plugin also fetches terminal size on client machine and sends it to pseudo terminal on target server.

ah, I think I get it. When I try the same command from a terminal with an enormous scrollback, it fails with Cannot perform start session: EOF. Using a new terminal session clears the scrollback, and aws ssm start-session works as expected.

[sruthi-maddineni]

Thanks for letting us know that you were able to get past this issue.
Please let us know in case of further issues.

[zengben-gainfully]

I had similar issue with "aws ecs execute-command ..." and it seems it's related to tty being not available when running in ci

I fixed it with "unbuffer aws ecs execute-command ..." to run it with tty available

@olfway You saved my day! I had the exact same issue and sudo apt-get install expect; aws ecs execute-command .... worked like a charm! Thanks.

[RichardTMiles]

More on unbuffer & expect
https://stackoverflow.com/questions/66066753/aws-start-session-end-with-cannot-perform-start-session-eof

Side note, you can run single commands like so

unbuffer aws ssm start-session \
         --document-name 'AWS-StartNonInteractiveCommand' \
         --parameters '{"command": ["sudo sh -c \"systemctl status boot_scripts && ( journalctl -u boot_scripts | cat )\""]}' \
         --target "$instanceid"

[wissnusetiawan]

maybe this command can help

• name: Install AWS CLI
  run: |
  sudo apt-get update
  sudo apt-get install -y awscli
  aws --version

• name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
  aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
  aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  aws-region: ${{ secrets.AWS_DEFAULT_REGION }}

• name: Install Session Manager Plugin
  run: |
  curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb" -o "session-manager-plugin.deb"
  sudo dpkg -i session-manager-plugin.deb
  sudo apt-get update
  sudo apt-get install -y session-manager-plugin
  sudo apt-get install -y expect
  session-manager-plugin --version

• name: Create Start SSH Session
  shell : sh
  run: |
  aws ssm start-session
  --target ${{ secrets.INSTANCE_ID }}
  --document-name AWS-StartSSHSession

[baguilarq]

Hello, any update about this?

[RichardTMiles]

@baguilarq, please read the entire thread and links before commenting. This ticket should be closed. I have posted a solution for this problem above.

[aries1980]

More on unbuffer & expect https://stackoverflow.com/questions/66066753/aws-start-session-end-with-cannot-perform-start-session-eof

Side note, you can run single commands like so

unbuffer aws ssm start-session \
         --document-name 'AWS-StartNonInteractiveCommand' \
         --parameters '{"command": ["sudo sh -c \"systemctl status boot_scripts && ( journalctl -u boot_scripts | cat )\""]}' \
         --target "$instanceid"

Can you elaborate how is this solve the issue? It doesn't for me and I have both unbuffer and expect available.
I can log in to most of the instances expect some with the error above and I can't grasp the difference.

[baguilarq]

More on unbuffer & expect https://stackoverflow.com/questions/66066753/aws-start-session-end-with-cannot-perform-start-session-eof
Side note, you can run single commands like so

unbuffer aws ssm start-session \
         --document-name 'AWS-StartNonInteractiveCommand' \
         --parameters '{"command": ["sudo sh -c \"systemctl status boot_scripts && ( journalctl -u boot_scripts | cat )\""]}' \
         --target "$instanceid"

Can you elaborate how is this solve the issue? It doesn't for me and I have both unbuffer and expect available. I can log in to most of the instances expect some with the error above and I can't grasp the difference.

did you try to do that without break lines?

[aries1980]

Thank you. The command that I tried to execute is what OP tried: aws ssm start-session --target i-xyz .

I added unbuffer to the beginning, same error: unbuffer aws ssm start-session --target i-xyz . My question was, why using unbuffer would make it work?

[RichardTMiles]

This question should be closed, as I've answered it already above.
Google and StackOverflow can help you with why a solution or command works. I recommend the spirit of exploration always. That being said, I feel nice, in the hope that no one else has questions about this.

Historically, the command aws ssm start-session is designed to start a direct connection to the target instance by means of a terminal (sh/bash/zsh/etc). Because of this aws ssm start-session requires a TTY to be available, which is not in CI/CD pipelines. The tty command of the terminal basically prints the file name of the terminal connected to standard input.

Thus aws ssm start-session is requesting access to standard input and does not have that keyboard access in a CI/CD pipeline context. We can mock this by using unbuffer. If a keyboard, or stdin, is actually needed to input real-time for a program, you can use the expect command, which is commonly used with unbuffer. I do not recommend this, however, as 99.9999% of the time, the program's built-in command line arguments will suffice.