Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make EKS fetcher to lazy initialize AWS EKS client #35077

Merged
merged 3 commits into from Dec 1, 2023

Conversation

AntonAM
Copy link
Contributor

@AntonAM AntonAM commented Nov 28, 2023

If one of EKS fetchers initialization encountered an error (for example non authorized to assume IAM role), it stopped the whole discovery process from starting, instead of just skipping this fetcher and logging the problem. This PR changes it so we log warning and continue initialization.

Changelog: Prevent EKS fetcher not having correct IAM permissions from stopping whole Discovery service start up

Fixes #34949

@AntonAM AntonAM added aws Used for AWS Related Issues. kubernetes discovery labels Nov 28, 2023
Copy link

The PR changelog entry failed validation: Changelog entry not found in the PR body. Please add a "no-changelog" label to the PR, or changelog lines starting with changelog: followed by the changelog entries for the PR.

@AntonAM AntonAM force-pushed the anton/eks-fetcher-error-skipping branch from 3607e72 to f22a9ef Compare November 28, 2023 20:47
Copy link

The PR changelog entry failed validation: Changelog entry not found in the PR body. Please add a "no-changelog" label to the PR, or changelog lines starting with changelog: followed by the changelog entries for the PR.

@AntonAM AntonAM force-pushed the anton/eks-fetcher-error-skipping branch from f22a9ef to 4da6fbc Compare November 28, 2023 20:54
Copy link

The PR changelog entry failed validation: Changelog entry not found in the PR body. Please add a "no-changelog" label to the PR, or changelog lines starting with changelog: followed by the changelog entries for the PR.

@AntonAM AntonAM force-pushed the anton/eks-fetcher-error-skipping branch from 4da6fbc to f286327 Compare November 28, 2023 21:13
Comment on lines +400 to +401
s.Log.WithError(err).Warnf("Could not initialize EKS fetcher(Region=%q, Labels=%q, AssumeRole=%q), skipping.", region, matcher.Tags, matcherAssumeRole.RoleARN)
continue
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this way, the issue where the Teleport service is unable to start will be mitigated, though the UX won't be perfect.

The EKS discovery with missing IAM permissions will be skipped, so if the missing IAM permission is updated, the discovery service requires a restart to apply this change.

Can do "lazy evaluation" when the EKS feature is created so we will allow to crate a EKS fetcher without full IAM permissions but the evolution will be done in the factual API call ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added lazy initialization for the EKS client, so now it will be retrying automatically on every Get.

@AntonAM AntonAM force-pushed the anton/eks-fetcher-error-skipping branch from 969dbba to 579d000 Compare November 30, 2023 20:00
@AntonAM AntonAM force-pushed the anton/eks-fetcher-error-skipping branch from 579d000 to 98c73f1 Compare November 30, 2023 21:53
@AntonAM
Copy link
Contributor Author

AntonAM commented Nov 30, 2023

@fspmarshall @rudream friendly ping

URL *url.URL
ARN string
URL *url.URL
AssumeRoleErrors map[string]error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is AssumeRoleErrors an accidental leftover from an earlier iteration of this work? I can't see anywhere that errors are ever actually added to this mapping outside of tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, we don't need it anymore after I changed EKS fetcher to lazy initialization. Removed.

@AntonAM AntonAM changed the title Continue discovery initialization after EKS fetcher error Make EKS fetcher to lazy initialize AWS EKS client Dec 1, 2023
@AntonAM AntonAM added this pull request to the merge queue Dec 1, 2023
Merged via the queue into master with commit e33830a Dec 1, 2023
36 checks passed
@AntonAM AntonAM deleted the anton/eks-fetcher-error-skipping branch December 1, 2023 19:57
@public-teleport-github-review-bot

@AntonAM See the table below for backport results.

Branch Result
branch/v12 Failed
branch/v13 Failed
branch/v14 Create PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

EKS Auto discovery missing IAM permission stops process execution
4 participants