Sync each app when fetched rather than waiting. #6825
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
force-pushed
the
6706-sync-public-oci-repo-4
branch
from
e753321
to
75f4f73
Compare
✅ Deploy Preview for kubeapps-dev canceled.
|
### Description of the change Currently ensures that the existing (already synced) chart versions are included when syncing new chart versions, so that the full set of data is retained in the DB (since the chart versions aren't actually rows in the db, but rather within a json blob of the chart). This PR refactored the way the syncing happens, specifically ensuring that data is synced to the db as each chart is processed, rather than when all charts are synced (so that we get better feedback in the UI when syincing OCI repos). ### Benefits Before the change: - a shallow + deep sync of the non-OCI Helm repo takes around 7s - a shallow sync of the Bitnami OCI repo would see us hitting the rate limit and getting 429s. With the changes, - a shallow + deep sync of a non-OCI Helm repo takes around 10s (extra overhead of sycing to the DB for each chart) - a shallow sync of the Bitnami OCI repo takes 2 minutes and a syncing the next 5 versions for each chart takes a further 2 minutes (and does not excessively hammer the dockerhub distribution spec API to get banned). (should remove the shallow and just always do 5, also should test 10). ### Possible drawbacks Currently: - ~~Something not correct with the db info being synced as getting `Unable to get the available package detail for the package "testoci2/airflow" using the plugin "helm.packages": internal: Unable to retrieve chart files: sql: no rows in result set`~~ Fixed - it was a synchronization issue that left the sync process finishing after charts were synced but before files were synced. - ~~OCI catalog is taking a *long* time to display (shouldn't be affected, getting info from the DB).~~ This was a resource issue locally. ~~So still more investigation here needed.~~ ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - fixes #6706 ### Additional information <!-- If there's anything else that's important and relevant to your pull request, mention that information here.--> --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
Signed-off-by: Michael Nelson <minelson@vmware.com>
Signed-off-by: Michael Nelson <minelson@vmware.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of the change
Follows on from #6804, this PR updates so that the actual Syncing happens per chart+versions, rather than at the end of the process. Much of the change is refactoring functions that worked with slices of charts to work with a chart instead. The complexity is the actual refactoring of the sync, requiring channels to iterate on the results etc.
Benefits
We'll be able no have results in the UX much more quickly (important with OCI imports, which take longer without a big index.yaml to simply parse).
Possible drawbacks
Currently this will not work as expected, as two more changes are required:
Applicable issues
Additional information
Last PR to follow.