-
Notifications
You must be signed in to change notification settings - Fork 400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
index.yaml regenerating every request even without any package updated #152
Comments
@IMBlues thanks for detailed issue and PR! Few things:
|
1 & 2. I can confirm this behavior because "Change detected between cache and storage" showed every request to The timestamp from storage(S3 backend, ceph rgw provides) has higher precision than from cache, so this comparison will always return true: if !o1.LastModified.Equal(o2.LastModified) {
diff.Updated = append(diff.Updated, o2)
} print them:
I have not tested it on Amazon S3, so I'm not sure from whether it's related to ceph rgw. |
@IMBlues sorry we left this unfinished. Were you able to determine if this is related to Ceph? |
No, I didn't test it on other s3 backend and this pr does solve my problem. I think we should close it now, I will reopen it if I have more time to determine . |
@IMBlues if the fix works for you, then we should add a config option that lets us get to the "ignore milliseconds" functionality. Maybe something like a You then use something like |
@jdolitsky I think adding flag is better than what I did(directly ignore milliseconds in code), maybe I'll try it later this week and make another pr |
Hello, We also see this problem when using Ceph as a backend for chartmuseum. |
We also suffered this issue with Dell ECS S3 storage backend. The previously provided PR by @IMBlues apparently has the Substraction operation reversed, as the o1 (in the cache) can never be newer than o2 (in the s3), so a new file will never be detected. I am providing a new PR with same fix but reversing the substraction order. |
Same issue with OTC S3 backend. I would love to see a fix in the upcoming release. Otherwise I have to build chartmuseum on my own, just because of ~3 lines of code. |
@IMBlues @mateuszkwiatkowski @airadier @jopfeffe Would one of you be able to test out master branch with new option You can use Docker image Once I can get confirmation that this is finally fixed (as well as #280), I'll go ahead and release chartmuseum v1.12.0. Thanks in advance! |
We are experiencing the same issue with our Openstack Swift based storage. The index is always rebuilt and takes around 30 seconds to complete every EDIT: |
@rajiteh - please see https://github.com/chartmuseum/storage/blob/master/amazon.go#L109 We will need to add a and yes, youre probably right about this ( |
It turns out the issue I had with openstack is fixed by this change: chartmuseum/storage@a49401b So my issue in particular should be resolved in 1.12.0 release. However, I already went ahead and implemented the timestamp argument. |
@rajiteh thank you very much, looking at your change |
Works in my case. Thank you very much for the effort. Looking forward for the 1.12.0 release. |
@jopfeffe thanks for trying this out! glad to hear |
Hi all, new version of chartmuseum has been released (v0.12.0) with a solution for this:
Big thanks to @rajiteh for putting this together! Thanks all |
The index.yaml will regenerate every request to
/index.yaml
even though there is no any package changed.When we compare the
LastModified
between objects from cache and storage:https://github.com/helm/chartmuseum/blob/master/pkg/storage/storage.go#L64
The objects from cache is out of order due to map ranging....
https://github.com/helm/chartmuseum/blob/master/pkg/chartmuseum/server/multitenant/index.go#L107
And the
LastModifed
timestamp of object from storage seems has higher precision (I only test it on s3 backend)Those above would cause the regenerating of index.yaml every request.
The text was updated successfully, but these errors were encountered: