Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retries to get ec2 instance-id #2861

Merged
merged 1 commit into from
May 5, 2021

Conversation

mythri-garaga
Copy link
Contributor

Summary

Currently when agent fails to get instance id from EC2 metadata service, agent overwrites InstanceID in state file with ''. This PR includes changes to call instance metadata service at least 3 times to get instance id before returning an empty string.

level=critical time=2021-03-04T00:00:05Z msg="Error saving state before final shutdown: Multiple error:\n\t0: final save: timed out trying to save to disk" module=termination_handler.go
level=info time=2021-03-04T00:00:07Z msg="Loading configuration" module=agent.go
level=info time=2021-03-04T00:00:07Z msg="Amazon ECS agent Version: 1.50.0, Commit: 90ccb1d4" module=agent.go
level=info time=2021-03-04T00:00:07Z msg="Image excluded from cleanup: amazon/amazon-ecs-pause:0.1.0" module=docker_image_manager.go
level=info time=2021-03-04T00:00:07Z msg="Image excluded from cleanup: amazon/amazon-ecs-pause:0.1.0" module=docker_image_manager.go
level=info time=2021-03-04T00:00:07Z msg="Image excluded from cleanup: amazon/amazon-ecs-agent:latest" module=docker_image_manager.go
level=info time=2021-03-04T00:00:07Z msg="Creating root ecs cgroup: /ecs" module=init_linux.go
level=info time=2021-03-04T00:00:07Z msg="Creating cgroup /ecs" module=cgroup_controller_linux.go
level=info time=2021-03-04T00:00:07Z msg="Event stream ContainerChange start listening..." module=eventstream.go
level=warn time=2021-03-04T00:00:07Z msg="Unable to access EC2 Metadata service to determine EC2 ID: RequestError: send request failed\ncaused by: Get \"http://169.254.169.254/latest/meta-data/instance-id\": EOF" module=agent.go
level=warn time=2021-03-04T00:00:07Z msg="Data mismatch; saved InstanceID 'i-xxxxxxxxx' does not match current InstanceID ''. Overwriting old datafile" module=agent.go
level=info time=2021-03-04T00:00:08Z msg="Registering Instance with ECS" module=agent.go

Implementation details

  • In `./agent/app/agent.go - Add retries with exponential back off when calling IMDS for ec2 instance id
  • In `./agent/app/agent_test.go - Updated tests accordingly

Testing

make test passes
New tests cover the changes: yes

Description for the changelog

Enhancement - Add retries while retrieving instance-id using EC2 Instance metadata service api

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

agent/app/agent.go Show resolved Hide resolved
agent/app/agent.go Outdated Show resolved Hide resolved
agent/app/agent.go Outdated Show resolved Hide resolved
agent/app/agent.go Outdated Show resolved Hide resolved
agent/app/agent.go Show resolved Hide resolved
sharanyad
sharanyad previously approved these changes May 3, 2021
sparrc
sparrc previously approved these changes May 3, 2021
Realmonia
Realmonia previously approved these changes May 3, 2021
agent/app/agent.go Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants