Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve source controller sync processing #94

Closed
nab-gha opened this issue Oct 1, 2020 · 0 comments
Closed

Improve source controller sync processing #94

nab-gha opened this issue Oct 1, 2020 · 0 comments

Comments

@nab-gha
Copy link
Contributor

nab-gha commented Oct 1, 2020

At present the mapping function that processes the gitrepository.source.toolkit.fluxcd.io custom resources gets the current repository contents from the source controller, identifies the addon layers that use it and creates a link to the addon layer's data directory then requeues the addon layers for processing.

This works but has some issues. The process of getting the repository data from the source controller and unpacking it into the local filesystem involves deleting the existing copy of the data to avoid deleted files being retained. This creates a window when the symbolic link used by the addon layers to access this data points to a non existent directory. This causes the addon layer to be marked as failed. The subsequent reprocessing of the addon layer will find the data and continue but this generates errors and status updates that are alarming for the user.

The repos package that is used to process the gitrepository.source.toolkit.fluxcd.io custom resources contains a Repos object that keeps track of repos. This contains entries for each revision of a repo. A better solution would be for the Repos to contain Repo objects that relate to gitrepository.source.toolkit.fluxcd.io custom resources and introduce a 'RepoRevision' object that relates to a particular git commit in the repo.

I also propose that the mapping function should just manage the Repos, adding, deleting Repo objects to the map in Repos and requeuing any addon layers that use that Repo.

We should add code to the addon layers reconciliation processing to look for the Repo they use in Repos and then look in the Repo for a RepoRevision matching the Repo's current revision. If this is not present it will be created and data retrieved from the source controller. Then the symbolic link for the addon layers data can be updated to point at the new revision or adjusted if need be to point at pre-existing directory, which will occur if multiple addon layers are using the same gitrepository.source.toolkit.fluxcd.io custom resource and one has created and updated a new revision.

@nab-gha nab-gha closed this as completed Nov 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant