-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use LRUCache for Application Manifests Cache #5108
Conversation
Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5108 +/- ##
==========================================
- Coverage 22.82% 22.82% -0.01%
==========================================
Files 409 409
Lines 43338 43343 +5
==========================================
Hits 9891 9891
- Misses 32668 32673 +5
Partials 779 779 ☔ View full report in Codecov by Sentry. |
Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 🙆♂️
* Use LRUCache for Application Manifests Cache Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> * Use not constant but config value Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> --------- Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com>
#5126 #5128 #5130 (#5132) * Register otel TracerProvider to send traces (#5029) * Register otel TracerProvider to send traces Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> * Bump gRPC version Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> * Upgrade google.golang.org/grpc Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> --------- Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Set fetch-depth to 0 to create correct patches during git cherry-pick as much as possible (#5096) Signed-off-by: Yoshiki Fujikane <ffjlabo@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Bump github.com/docker/docker from 24.0.9+incompatible to 26.1.4+incompatible (#5097) * Bump github.com/docker/docker Bumps [github.com/docker/docker](https://github.com/docker/docker) from 24.0.9+incompatible to 26.1.4+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v24.0.9...v26.1.4) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Manually update docker/cli to pass the tests build errors Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: khanhtc1202 <khanhtc1202@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Bump postcss from 7.0.39 to 8.4.40 in /docs (#5099) Bumps [postcss](https://github.com/postcss/postcss) from 7.0.39 to 8.4.40. - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@7.0.39...8.4.40) --- updated-dependencies: - dependency-name: postcss dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Use LRUCache for Application Manifests Cache (#5108) * Use LRUCache for Application Manifests Cache Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> * Use not constant but config value Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> --------- Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Fix UI dependecies deprecated (#5113) Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Bump postcss and autoprefixer in /docs (#5114) Bumps [postcss](https://github.com/postcss/postcss) to 8.4.40 and updates ancestor dependency [autoprefixer](https://github.com/postcss/autoprefixer). These dependencies need to be updated together. Updates `postcss` from 7.0.39 to 8.4.40 - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@7.0.39...8.4.40) Updates `autoprefixer` from 9.8.8 to 10.4.20 - [Release notes](https://github.com/postcss/autoprefixer/releases) - [Changelog](https://github.com/postcss/autoprefixer/blob/main/CHANGELOG.md) - [Commits](postcss/autoprefixer@9.8.8...10.4.20) --- updated-dependencies: - dependency-name: postcss dependency-type: indirect - dependency-name: autoprefixer dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * fix: upgrade google-protobuf from 3.21.0 to 3.21.4 (#5115) Snyk has created this PR to upgrade google-protobuf from 3.21.0 to 3.21.4. See this package in yarn: google-protobuf See this project in Snyk: https://app.snyk.io/org/pipecd/project/f41c5767-b506-4f59-beb9-ef662258eb9a?utm_source=github&utm_medium=referral&page=upgrade-pr Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * fix: upgrade react-markdown from 6.0.2 to 6.0.3 (#5116) Snyk has created this PR to upgrade react-markdown from 6.0.2 to 6.0.3. See this package in yarn: react-markdown See this project in Snyk: https://app.snyk.io/org/pipecd/project/f41c5767-b506-4f59-beb9-ef662258eb9a?utm_source=github&utm_medium=referral&page=upgrade-pr Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * fix: tool/actions-gh-release/Dockerfile to reduce vulnerabilities (#5118) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-ALPINE320-OPENSSL-7413532 - https://snyk.io/vuln/SNYK-ALPINE320-OPENSSL-7413532 Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * fix: upgrade dayjs from 1.8.28 to 1.11.12 (#5126) Snyk has created this PR to upgrade dayjs from 1.8.28 to 1.11.12. See this package in yarn: dayjs See this project in Snyk: https://app.snyk.io/org/pipecd/project/f41c5767-b506-4f59-beb9-ef662258eb9a?utm_source=github&utm_medium=referral&page=upgrade-pr Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Bump github.com/docker/docker (#5128) Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Update RELEASE to v0.48.5 (#5130) Signed-off-by: Yoshiki Fujikane <ffjlabo@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> --------- Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> Signed-off-by: Yoshiki Fujikane <ffjlabo@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: khanhtc1202 <khanhtc1202@gmail.com> Co-authored-by: Khanh Tran <32532742+khanhtc1202@users.noreply.github.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Chris Aniszczyk <caniszczyk@gmail.com>
* Use LRUCache for Application Manifests Cache Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> * Use not constant but config value Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> --------- Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com>
) * Set fetch-depth to 0 to create correct patches during git cherry-pick as much as possible (#5096) Signed-off-by: Yoshiki Fujikane <ffjlabo@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Bump postcss from 7.0.39 to 8.4.40 in /docs (#5099) Bumps [postcss](https://github.com/postcss/postcss) from 7.0.39 to 8.4.40. - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@7.0.39...8.4.40) --- updated-dependencies: - dependency-name: postcss dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Use LRUCache for Application Manifests Cache (#5108) * Use LRUCache for Application Manifests Cache Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> * Use not constant but config value Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> --------- Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Fix UI dependecies deprecated (#5113) Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * Bump postcss and autoprefixer in /docs (#5114) Bumps [postcss](https://github.com/postcss/postcss) to 8.4.40 and updates ancestor dependency [autoprefixer](https://github.com/postcss/autoprefixer). These dependencies need to be updated together. Updates `postcss` from 7.0.39 to 8.4.40 - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@7.0.39...8.4.40) Updates `autoprefixer` from 9.8.8 to 10.4.20 - [Release notes](https://github.com/postcss/autoprefixer/releases) - [Changelog](https://github.com/postcss/autoprefixer/blob/main/CHANGELOG.md) - [Commits](postcss/autoprefixer@9.8.8...10.4.20) --- updated-dependencies: - dependency-name: postcss dependency-type: indirect - dependency-name: autoprefixer dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * fix: upgrade google-protobuf from 3.21.0 to 3.21.4 (#5115) Snyk has created this PR to upgrade google-protobuf from 3.21.0 to 3.21.4. See this package in yarn: google-protobuf See this project in Snyk: https://app.snyk.io/org/pipecd/project/f41c5767-b506-4f59-beb9-ef662258eb9a?utm_source=github&utm_medium=referral&page=upgrade-pr Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * fix: upgrade react-markdown from 6.0.2 to 6.0.3 (#5116) Snyk has created this PR to upgrade react-markdown from 6.0.2 to 6.0.3. See this package in yarn: react-markdown See this project in Snyk: https://app.snyk.io/org/pipecd/project/f41c5767-b506-4f59-beb9-ef662258eb9a?utm_source=github&utm_medium=referral&page=upgrade-pr Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * fix: tool/actions-gh-release/Dockerfile to reduce vulnerabilities (#5118) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-ALPINE320-OPENSSL-7413532 - https://snyk.io/vuln/SNYK-ALPINE320-OPENSSL-7413532 Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> * fix: upgrade dayjs from 1.8.28 to 1.11.12 (#5126) Snyk has created this PR to upgrade dayjs from 1.8.28 to 1.11.12. See this package in yarn: dayjs See this project in Snyk: https://app.snyk.io/org/pipecd/project/f41c5767-b506-4f59-beb9-ef662258eb9a?utm_source=github&utm_medium=referral&page=upgrade-pr Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> --------- Signed-off-by: Yoshiki Fujikane <ffjlabo@gmail.com> Signed-off-by: pipecd-bot <pipecd.dev@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com> Signed-off-by: t-kikuc <tkikuchi07f@gmail.com> Co-authored-by: Yoshiki Fujikane <40124947+ffjlabo@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shinnosuke Sawada-Dazai <shin@warashi.dev> Co-authored-by: Khanh Tran <32532742+khanhtc1202@users.noreply.github.com> Co-authored-by: snyk-bot <snyk-bot@snyk.io> Co-authored-by: Chris Aniszczyk <caniszczyk@gmail.com>
What this PR does / why we need it:
Use LRUCache for the application manifests cache instead of TTLCache to control memory consumption.
Investigation details
The provided piped CPU/memory profiles are below.
Memory Profile
CPU Profile
The CPU profiles shows garbage collection takes times, so there seems a lot of long-living objects.
The memory profiles show that the drift detector of Kubernetes makes many memory allocations.
The allocated objects in
kubernetes.(*loader).loadHeadManifests
are stored in the application manifests cache, and the cache key contains the git commit hash.pipecd/pkg/app/piped/driftdetector/kubernetes/detector.go
Lines 285 to 290 in f5dcaa4
This cache is memorystore.TTLCache, so the cache eviction occurs only in the TTL mechanisms.
When the commit hash is changed many times quickly, the cache grows larger and larger.
This PR changes the cache from TTLCache to LRUCache, so the cache size is now limited.
Which issue(s) this PR fixes:
N/A
Does this PR introduce a user-facing change?: