Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

helm/v3: garbage collect anonymous index files #422

Merged
merged 1 commit into from
May 21, 2020
Merged

Conversation

hiddeco
Copy link
Member

@hiddeco hiddeco commented May 20, 2020

The Helm 3 dependency update generates anonymous index files on every
run without garbage collecting them, resulting in spurious disk usage.
Configuring a temporary cache directory and garbage collecting it after
the dependency update is not possible as the download manager seems to
ignore the configured repository cache path and always create the index
file in the XDG cache directory.

Instead we detect the any anonymous index filex by their "=" suffix and
delete if they are older than 2m.

@hiddeco hiddeco added the bug Something isn't working label May 20, 2020
@hiddeco hiddeco requested a review from stefanprodan May 20, 2020 23:07
@hiddeco hiddeco force-pushed the helm/gc-dep-update branch 2 times, most recently from baa18ad to fe9e671 Compare May 21, 2020 08:27
Copy link
Member

@squaremo squaremo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments 💬!

pkg/helm/v3/dependency.go Outdated Show resolved Hide resolved
pkg/helm/v3/dependency.go Outdated Show resolved Hide resolved
@hiddeco hiddeco force-pushed the helm/gc-dep-update branch 2 times, most recently from 307f98b to 05f2b82 Compare May 21, 2020 08:57
Copy link
Member

@squaremo squaremo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🍇

pkg/helm/v3/dependency.go Show resolved Hide resolved
pkg/helm/v3/dependency.go Outdated Show resolved Hide resolved
The Helm 3 dependency update generates anonymous index files on every
run without garbage collecting them, resulting in spurious disk usage.
Configuring a temporary cache directory and garbage collecting it after
the dependency update is not possible as the download manager seems to
ignore the configured repository cache path and always create the index
file in the XDG cache directory.

Instead we detect the any anonymous index filex by their "=" suffix and
delete if they are older than 2m.
@hiddeco hiddeco merged commit 1d9f181 into master May 21, 2020
@hiddeco hiddeco deleted the helm/gc-dep-update branch May 21, 2020 09:37
@hiddeco hiddeco added this to the 1.1.0 milestone May 21, 2020
@ferrandinand
Copy link

I updated to helm-operator 1.1.0 but this problem is still there as -index.yaml and -charts.txt are not removed.

/ # du -h -s /root/.cache/helm/repository
10.1G   /root/.cache/helm/repository
  flux-helm-operator:
    Container ID:  docker://b4457b2bf530ab1871441c573d3804ad146946338838f4639287f2a75119d40c
    Image:         docker.io/fluxcd/helm-operator:1.0.1
    Image ID:      docker-pullable://fluxcd/helm-operator@sha256:a2a8bf7538458291ebde17a1f9e5ec0d711f6c9f0a3011e6eb9da680e5b82f61
    Port:          3030/TCP
    Host Port:     0/TCP
    Args:
      --enabled-helm-versions=v3
      --log-format=fmt
      --git-timeout=20s
      --git-poll-interval=5m
      --charts-sync-interval=3m
      --status-update-interval=30s
      --update-chart-deps=true
      --log-release-diffs=false
      --workers=4
      --tiller-namespace=kube-system
/ # ls -la /root/.cache/helm/repository
total 10576636
-rw-r--r--    1 root     root          3370 Jun  3 16:32 +1EK02SqaEAwVsIXA3tl8lPK64s=-charts.txt
-rw-r--r--    1 root     root       8493397 Jun  3 16:32 +1EK02SqaEAwVsIXA3tl8lPK64s=-index.yaml
-rw-r--r--    1 root     root          3370 Jun  3 17:28 +7TDZRyYcW3jM1iRy-awgDNFAbs=-charts.txt
-rw-r--r--    1 root     root       8493388 Jun  3 17:28 +7TDZRyYcW3jM1iRy-awgDNFAbs=-index.yaml
-rw-r--r--    1 root     root          3370 Jun  4 09:44 +81k-ha-3bQVj-DLN-2sD37Ucmk=-charts.txt
-rw-r--r--    1 root     root       8497898 Jun  4 09:44 +81k-ha-3bQVj-DLN-2sD37Ucmk=-index.yaml
-rw-r--r--    1 root     root          3370 Jun  3 17:26 +BDESiuNv-fjf29DQYP2mb-MhTY=-charts.txt
-rw-r--r--    1 root     root       8493388 Jun  3 17:26 +BDESiuNv-fjf29DQYP2mb-MhTY=-index.yaml
-rw-r--r--    1 root     root          3370 Jun  4 01:20 +DXau1YTZbBln-J7Z-HmErmGCkU=-charts.txt
-rw-r--r--    1 root     root       8495036 Jun  4 01:20 +DXau1YTZbBln-J7Z-HmErmGCkU=-index.yaml
-rw-r--r--    1 root     root          3370 Jun  4 08:01 +HmzFV50LLmf0uyxUxWC6wEcVPo=-charts.txt
[OMITED FULL OUTPUT]

@hiddeco
Copy link
Member Author

hiddeco commented Jun 4, 2020

@ferrandinand

  flux-helm-operator:
    Container ID:  docker://b4457b2bf530ab1871441c573d3804ad146946338838f4639287f2a75119d40c
    Image:         docker.io/fluxcd/helm-operator:1.0.1

Clearly states you are still running 1.0.1?

@ferrandinand
Copy link

me 👎.

Thanks @hiddeco

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants