Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws ssm start-session fails when behind HTTP proxy #9282

Open
1 task
rbeede opened this issue Feb 6, 2025 · 1 comment
Open
1 task

aws ssm start-session fails when behind HTTP proxy #9282

rbeede opened this issue Feb 6, 2025 · 1 comment
Assignees
Labels
bug This issue is a bug. p3 This is a minor priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. ssm

Comments

@rbeede
Copy link

rbeede commented Feb 6, 2025

Describe the bug

The CLI fails to operate correctly when behind an HTTP proxy. This is NOT for the ssm-agent but on the calling client side. The AWS CLI does not work for making wss:// protocol connections behind an HTTP proxy such as Burp Suite.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

The wss:// traffic to appear in the HTTP proxy software (Burp Suite) and the terminal to function as normal.

Current Behavior

  1. Command begins with "Starting session with SessionId: session_id_here"
  2. HTTP request and response are seen in proxy (Burp Suite)
  3. Terminal hangs - unresponsive to ctrl+C, only kill command terminates

Reproduction Steps

Start an HTTP proxy such as Burp Suite.

Open a terminal and paste the following env. variables:

export https_proxy=http://localhost:8080
export HTTPS_PROXY=http://127.0.0.1:8080
export HTTP_PROXY=http://127.0.0.1:8080
export http_proxy=http://localhost:8080
export ALL_PROXY=http://localhost:8080
export AWS_CA_BUNDLE=/home/username/Documents/burp-ca.pem
export all_proxy=http://localhost:8080

In that same terminal attempt to start an SSM session:

aws ssm start-session \
    --target i-0000000000 \
    --region us-east-1 \
    --profile MyProfileWithPermissions

One HTTP request + response is seen in the proxy, but no WebSocket traffic. The terminal also hangs and is unresponsive as described in the Current Behavior section.

Possible Solution

Workaround:

export NO_PROXY=ssmmessages.us-east-1.amazonaws.com

The terminal will work, but no wss:// traffic will be captured in the Burp Suite proxy.

Additional Information/Context

Adding aws --no-verify-ssl ssm start-session does not help either.

CLI version used

aws-cli/2.19.1 Python/3.12.6 Linux/6.8.0-51-generic exe/x86_64.ubuntu.22

Environment details (OS name and version, etc.)

Ubuntu 22.04

@rbeede rbeede added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 6, 2025
@adev-code adev-code self-assigned this Mar 10, 2025
@adev-code adev-code added investigating This issue is being investigated and/or work is in progress to resolve the issue. ssm p3 This is a minor priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Mar 10, 2025
@adev-code
Copy link

Hi @rbeede , the CLI flow shows that the StartSession request reaches ssm.us-east-1.amazonaws.com and gets a 200 OK response, returning a StreamUrl pointing to wss://ssmmessages.us-east-1.amazonaws.com. This is where the CLI stops using HTTPS and upgrades to WebSockets. If the session hangs or fails at this point, it’s because the WebSocket connection isn't establishing properly.

Confirming that bypassing the proxy with NO_PROXY fixes it, this means the proxy is either blocking WebSockets entirely or modifying something in the connection request that breaks. If the proxy is modifying headers or tampering with the request in any way, AWS will reject it. If the proxy just doesn’t support WebSockets properly, the connection won’t establish at all.

To debug, check if WebSocket traffic is actually passing through their proxy and if the Upgrade request is making it to AWS intact. If it isn’t, then the proxy is blocking it outright. If it is, but the connection still fails, then something is being altered that breaks AWS’s expected request format. Optionally, capture network traffic at the point where the WebSocket connection is attempted and compare it to what the CLI does when NO_PROXY is set. That will show exactly what’s different. Please let me know for other questions thanks.

@adev-code adev-code added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Mar 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p3 This is a minor priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. ssm
Projects
None yet
Development

No branches or pull requests

2 participants