Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Completely ignore Instance Metadata when in SQS Queue mode. #735

Merged
merged 5 commits into from
Dec 6, 2022
Merged

Completely ignore Instance Metadata when in SQS Queue mode. #735

merged 5 commits into from
Dec 6, 2022

Conversation

peteroruba
Copy link
Contributor

Issue #, if available:
Fixes #732

When IMDSv2 is enabled and aws-note-termination-handler is ran inside a container accessing IMDSv2 is not possible due to the IMDSv2 HTTP endpoint's default hop limit of 1. This makes the process exit.

Description of changes:
Completely disables the communication with the IMDS endpoint when in SQS queue mode.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@peteroruba peteroruba requested a review from a team as a code owner November 30, 2022 12:17
Comment on lines 353 to 364
} else {
metadata.AccountId = ""
metadata.InstanceID = ""
metadata.InstanceLifeCycle = ""
metadata.InstanceType = ""
metadata.PublicHostname = ""
metadata.PublicIP = ""
metadata.LocalHostname = ""
metadata.LocalIP = ""
metadata.AvailabilityZone = ""
metadata.Region = ""
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this else branch necessary? When imdsDisabled=false, metadata should still be empty, since no properties were set. When you were testing, did you discover this was required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

He complained about metadata being uninitialized so I've set it to at least empty string values.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who is "he"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The aws-node-termination-handler process. I ran it both on AWS EKS and on my workstation's local k8s cluster.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right. If you replace line 329 with the following, it will initialize the whole struct and you won't need to set each property inidividually:

metadata := NodeMetadata{}

Copy link
Contributor

@snay2 snay2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This unit test needs an update: https://github.com/aws/aws-node-termination-handler/blob/main/pkg/ec2metadata/ec2metadata_test.go#L583

Also, if possible, please add a new unit test to verify that IMDS does not get called when calling GetNodeMetadata() in Queue Processor mode.


imds := ec2metadata.New(nthConfig.MetadataURL, nthConfig.MetadataTries)

interruptionEventStore := interruptioneventstore.New(nthConfig)
nodeMetadata := imds.GetNodeMetadata()
nodeMetadata := imds.GetNodeMetadata(!imdsDisabled)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that both the variable and the function parameter are called "disabled", shouldn't this line be the following?

Suggested change
nodeMetadata := imds.GetNodeMetadata(!imdsDisabled)
nodeMetadata := imds.GetNodeMetadata(imdsDisabled)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, good catch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify: GetNodeMetadata() does get called in queue mode with the current code as per this MR. Unfortunately my Gelang skills are not sufficient to come up with an elegant refactoring that truly separates the code parts.

@peteroruba
Copy link
Contributor Author

Unfortunately I am not able to run the tests locally, as localstack doesn't come up properly:

🥑 Using localstack pod localstack-7f5f94f966-qbdr5

Could not connect to the endpoint URL: "http://localhost:4597/"
command terminated with exit code 255
error: You must provide one or more resources by argument or filename.
Example resource specifications include:
'-f rsrc.yaml'
'--filename=rsrc.json'
' '
''
⏰ Took 385sec
❌ NTH Integration Test FAILED e2e-test-bed7523d! ❌
🥑 Deleting k8s cluster using "kind"
Deleting cluster "e2e-test-bed7523d" ...
make: *** [e2e-test] Error 1

Copy link
Contributor

@snay2 snay2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three changes still needed:

  • Update the NodeMetadata initialization logic (see this comment)
  • Add a unit test for GetNodeMetadata() when IMDS is disabled
  • Update the README to indicate that we no longer query IMDS on startup when using Queue Processor mode

I made these changes in a commit on my fork: snay2@d2b2256

You can cherry-pick that commit onto your branch and update the PR. Otherwise, I can push the commit directly to your branch, whichever you prefer.

Comment on lines 353 to 364
} else {
metadata.AccountId = ""
metadata.InstanceID = ""
metadata.InstanceLifeCycle = ""
metadata.InstanceType = ""
metadata.PublicHostname = ""
metadata.PublicIP = ""
metadata.LocalHostname = ""
metadata.LocalIP = ""
metadata.AvailabilityZone = ""
metadata.Region = ""
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right. If you replace line 329 with the following, it will initialize the whole struct and you won't need to set each property inidividually:

metadata := NodeMetadata{}

peteroruba and others added 2 commits December 6, 2022 07:29
Update README to reflect new behavior. Fix NodeMetadata initialization logic.
@peteroruba
Copy link
Contributor Author

@snay2 I cherry-picked your commit. Your assistance on this matter is much appreciated.

Copy link
Contributor

@snay2 snay2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Crash with exit code 2 in Queue Processing mode and IMDSv2 enabled on EC2
2 participants