Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add cached k8s client #84

Merged
merged 11 commits into from
Aug 21, 2023
Merged

feat: Add cached k8s client #84

merged 11 commits into from
Aug 21, 2023

Conversation

MikeGoldsmith
Copy link
Contributor

@MikeGoldsmith MikeGoldsmith commented Aug 18, 2023

Which problem is this PR solving?

Uses the kubernetes client directly to query resource details (eg pod, service and nodes) is both expensive and inefficient because it does network requests to the cluster each time. This also led to delays in processing events because the cluster API would enforce backoffs to repeated frequent requests.

This PR introduces a cached client using informers which sets up watchers for the resources we care about and creates in-memory caches for us to query instead.

Short description of the changes

  • Create utils.CachedK8sClient
  • Setup and start the cache in main.go
  • Replace direct usage of k8s client with new cached version instead
  • Re-enable k8s metadata attrs when processing events
  • Refactor k8s metadata function to only add metadata for resources it can find (pod, service, etc)

How to verify that this has the expected result

  • k8s metadata is now visible on sent events
  • There is no slow-down of events because of API query backoffs

@MikeGoldsmith MikeGoldsmith added the type: enhancement New feature or request label Aug 18, 2023
@vreynolds vreynolds added this to the Trial with HNY Dogfood milestone Aug 18, 2023
@MikeGoldsmith MikeGoldsmith marked this pull request as ready for review August 21, 2023 11:05
@MikeGoldsmith MikeGoldsmith requested a review from a team August 21, 2023 11:05
Co-authored-by: Purvi Kanal <kanal.purvi@gmail.com>
Copy link
Contributor

@vreynolds vreynolds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k8s attributes are happening without increase in event handled latency 🌮

@vreynolds vreynolds merged commit 57028ac into main Aug 21, 2023
3 checks passed
@vreynolds vreynolds deleted the mike/cached-k8s branch August 21, 2023 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: optimize k8s client
3 participants