SSM Agent fails to start on latest alpha

[misterorion]

Description

I'm testing out the latest Alpha (3815.0.0) on EC2 and I see that the amazon-ssm-agent service is throwing an error and is not able to start. Basically:

amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory

Impact

The impact is not high since rolling back to the latest Stable AMI (3602.2.3) does not produce the error, but I wanted to make this visible somewhere.

Environment and steps to reproduce

1. Set-up: Arm64 EC2 instance running Alpha (3815.0.0) and a few docker containers via systemd. All EC2 instance roles are configured correctly to allow the SSM agent to connect.
2. Task: After launching the test instance, I tried to execute some SSM documents and noticed that no managed instance targets were found (because the SSM agent wasn't connected).
3. Action(s): In my EC2 launch-template, I simply swapped Stable AMI with Alpha AMI and re-launched the instance.
4. Error: I see multiple errors related to identity_config.json. See below for some examples. The systemd service does not start and enters a failed state after some time.

amazon-ssm-agent[8929]: 2023/12/28 15:29:32 Found config file at /etc/amazon/ssm/amazon-ssm-agent.json.
amazon-ssm-agent[8929]: Applying config override from /etc/amazon/ssm/amazon-ssm-agent.json.
amazon-ssm-agent[8929]: 2023/12/28 15:29:32 processing appconfig overrides
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 INFO [ssm-agent-worker] Checking if agent identity type CustomIdentity can be assumed
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 ERROR [ssm-agent-worker] Agent failed to assume any identity
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 ERROR [ssm-agent-worker] failed to find identity, retrying: failed to find agent identity
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 INFO [ssm-agent-worker] Checking if agent identity type OnPrem can be assumed
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 INFO [ssm-agent-worker] Checking if agent identity type EC2 can be assumed
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 INFO [ssm-agent-worker] Checking if agent identity type CustomIdentity can be assumed
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 WARN [ssm-agent-worker] failed to read runtime config 'identity_config.json': open /var/lib/amazon/ssm/runtimeconfig/identity_config.json: no such file or directory
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 ERROR [ssm-agent-worker] Agent failed to assume any identity
amazon-ssm-agent[8929]: 2023-12-28 15:29:32 ERROR [ssm-agent-worker] error occurred when starting ssm-agent-worker: failed to find agent identity
systemd[1]: amazon-ssm-agent.service: Deactivated successfully.

Expected behavior

The amazon-ssm-agent service should start normally and the agent should connect.

Additional information

The only change I made to my configuration was the Flatcar version (AMI) for my region. As I mentioned, with the latest Stable version, everything works perfectly. I'm not sure what additional information to provide, but I am available to provide more if needed.

[tormath1]

Hello @misterorion and sorry for the delay, most of the team was AFK for the end of the year. I'm checking this issue and I think it's more a configuration issue that might be caused by this Amazon SSM Agent bump version here: flatcar/scripts@0774334

Where it goes from 2.3.1319.0 to 3.2.985.0. I'm trying to understand what's missing to help you and to document it.

[tormath1]

To be honest, I'm a bit stuck on this one - opened an issue on the upstream to get some help / input: aws/amazon-ssm-agent#554

[misterorion]

Thanks @tormath1 for looking into this one. I Googled the error and found this issue with a similar error: aws/amazon-ssm-agent#515

Seems like there was a path change somewhere along the line for some OSes. Maybe this can give us a clue?

[tormath1]

Hello @misterorion, thanks for raising the issue - it should now work on Alpha, Beta (and Stable). Let us know!

[misterorion]

Hi @tormath1, I updated to the latest AMI and see no issues with the SSM agent. Thanks for getting this fixed!