[Issue]failed to find agent identity

[edfenergy-robcarter]

Trying to set this up to connect our local VSCode instances to our sagemaker studio instances for better developer experience.

When running:
%%sh
sm-ssh-ide ssm-agent

We receive the following error:

Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelog.xml: no such file or directory
Initializing new seelog logger
New Seelog Logger Creation Complete
2023-06-16 10:55:53 INFO Proxy environment variables:
2023-06-16 10:55:53 INFO https_proxy: 
2023-06-16 10:55:53 INFO http_proxy: 
2023-06-16 10:55:53 INFO no_proxy: 
2023-06-16 10:55:53 INFO Checking if agent identity type OnPrem can be assumed
2023-06-16 10:55:53 INFO Checking if agent identity type EC2 can be assumed
2023-06-16 10:55:53 ERROR [EC2Identity] failed to get identity instance id. Error: EC2MetadataError: failed to get IMDSv2 token and fallback to IMDSv1 is disabled
caused by: : 
	status code: 0, request id: 
caused by: RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: invalid argument
2023-06-16 10:55:53 INFO Checking if agent identity type CustomIdentity can be assumed
2023-06-16 10:55:53 ERROR Agent failed to assume any identity
2023-06-16 10:55:53 ERROR failed to find identity, retrying: failed to find agent identity
2023-06-16 10:55:53 INFO Checking if agent identity type OnPrem can be assumed
2023-06-16 10:55:53 INFO Checking if agent identity type EC2 can be assumed
2023-06-16 10:55:54 ERROR [EC2Identity] failed to get identity instance id. Error: EC2MetadataError: failed to get IMDSv2 token and fallback to IMDSv1 is disabled
caused by: : 
	status code: 0, request id: 
caused by: RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: invalid argument
2023-06-16 10:55:54 INFO Checking if agent identity type CustomIdentity can be assumed
2023-06-16 10:55:54 ERROR Agent failed to assume any identity
2023-06-16 10:55:54 ERROR failed to find identity, retrying: failed to find agent identity
2023-06-16 10:55:54 INFO Checking if agent identity type OnPrem can be assumed
2023-06-16 10:55:54 INFO Checking if agent identity type EC2 can be assumed
2023-06-16 10:55:54 ERROR [EC2Identity] failed to get identity instance id. Error: EC2MetadataError: failed to get IMDSv2 token and fallback to IMDSv1 is disabled
caused by: : 
	status code: 0, request id: 
caused by: RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: connect: invalid argument
2023-06-16 10:55:54 INFO Checking if agent identity type CustomIdentity can be assumed
2023-06-16 10:55:54 ERROR Agent failed to assume any identity
2023-06-16 10:55:54 ERROR failed to get identity: failed to find agent identity
2023-06-16 10:55:54 ERROR Error occurred when starting amazon-ssm-agent: failed to get identity: failed to find agent identity

[ivan-khvostishkov]

Hi, @edfenergy-robcarter , thank you for sharing the logs. Which version of the library do you use? You can check by typing pip freeze | grep sagemaker-ssh-helper .

You're probably using the old version, because some of the issues like #13 has been already fixed.

If you are on the newest version which is v1.11.0, please share which instance type and which SageMaker Studio kernel you're running this on.

[edfenergy-robcarter]

Hi Ivan,
This is a first time setup, so we're not attached to any version yet.
Can confirm we're using the latest version 1.11.0 for the first attempt, but error message persists. Should we attempt to connect from local environment anyway? Issue #13 implies that it wouldn't be blocking.

The instance type is ml.t3.medium and the kernel is named as Python 3, (specifically python 3.8.16). We're constrained to python 3.8 for now because of other dependencies.

[ivan-khvostishkov]

Hi, @edfenergy-robcarter , thanks for confirming. Which SageMaker Studio image do you use this kernel in? Is it your custom-build image?

SageMaker SSH Helper has been tested and proved to be working in a number of standard kernels and images: https://github.com/aws-samples/sagemaker-ssh-helper/blob/v1.11.0/tests/test_ide.py#L18_L78

I would appreciate if you can try it in one of these images, e.g. PyTorch 1.12 Python 3.8 CPU Optimized, which is also Python 3.8.16, and let me know the results. In case it works for you with the standard pre-built image and doesn't work in your custom image, we might need to look closer into this difference.

[edfenergy-robcarter]

Hi, I have tried in the image you recommend, we got this error:

An error occurred (AccessDeniedException) when calling the CreateActivation operation: User: arn:aws:sts::<account_number>:assumed-role/role-cus-data-science-sagemaker-pri/SageMaker is not authorized to perform: ssm:CreateActivation on resource: arn:aws:iam::<account_number>:role/role-cus-data-science-sagemaker-pri because no identity-based policy allows the ssm:CreateActivation action

I am assuming this means we need to change our permissions, I will consult DevOps.

[ivan-khvostishkov]

Hi, @edfenergy-robcarter , you are right that you need additional IAM/SSM setup from your admins. This requirement is mentioned in the section Getting started, which refers to the page Setting up your AWS account with IAM and SSM configuration with the necessary configuration.

Let me know if you managed to make it work?

[ivan-khvostishkov]

I will close the issue for now.
@edfenergy-robcarter Please, reopen, when you need further support.