Optimise clone operations #665
Conversation
|
Have been thinking some more about putting this behind a feature flag @pjbgf. False-positives would not be that much of an issue, as we are optimizing :-). But locking in unexpected ways (and thereby not moving forward anymore), would be an issue. (Although by looking at the code, this does not seem very plausible.) But to better be safe than sorry, let's feature flag it for at least one minor range. Then review as follow up on next next minor. I have been thinking about the growth of feature flags, which then later need to be removed (and deprecated) from the arguments. For the Git improvement, we used an environment variable for this. However, I am wondering we couldn't (also) add a flag accepting an array of items. We could then iterate over the array, enabling the feature flags for that release. Any flag that is no longer marked as experimental can just stops being detected (but logged to encourage removal), without breaking the further Deployment if the user does not update their config. This would allow for a less verbose declarative configuration, as environment variables in YAML are lengthy and nested. |
hiddeco
added
enhancement
We could use the same convention as Kubernetes uses for feature gates https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/ |
No-op reconciliations are very inefficient, as they carry out a full clone operation of the target repository even when no changes have taken place. This change will execute a remote-ls operation, and cancel the clone operation if the remote tip commit is still the same as the one observed on the last reconcilation. In such cases, an git.NoChangesError is returned. Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Signed-off-by: Sanskar Jaiswal <jaiswalsanskar078@gmail.com>
…luxcd#694) * Check if revision has changed in gogit CheckoutBranch Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
getBlankRepoAndRemote's callers are responsible for the disposal of the returned objects. However, the caller does not expect to need to dispose objects when err != nil, which may result to memory leaks. Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
|
Added the feature-gate and done some other housekeeping bits. PTAL |
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
|
I think it should be |
@stefanprodan the feature gate is managed via |
OptimizedGitClones decreases resource utilization for GitRepository reconciliations. It supports both go-git and libgit2 implementations when cloning repositories using branches or tags. This is an opt-out feature, which can be disabled by starting the controller with the argument '--feature-gates=OptimizedGitClones=false'. Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
stefanprodan
removed
the
experimental
|
Bonus for using feature gates! 💪🏻 |
Signed-off-by: Paulo Gomes <paulo.gomes@weave.works>
|
@aryan9600 @somtochiama @darkowlzz thank you very much for all the help getting this over the line. 🙇 |
No-op reconciliations are very inefficient, as they carry out a full (or shallow for
go-git) clone of the target repository even when no changes have taken place.The PR aims to rather optimise the process by first checking whether the last revision is still the same at the target repository, and if that is so, short-circuit the reconciliation. If not, reuse the same connection to fetch and reset the repository to the upstream
HEAD.This is an opt-out feature, which can be disabled by starting the controller with the argument
--feature-gates=OptimizedGitClones=false.Remaining tasks: