Check ignore matches before Bucket item downloads #337
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Hidde Beydals <hello@hidde.co>
This commit makes the filtering applied during the archiving configurable by introducing an optional `ArchiveFileFilter` callback argument and a `SourceIgnoreFilter` implementation. `SourceIgnoreFilter` filters out files matching sourceignore.VCSPatterns and any of the provided patterns. If an empty gitignore.Pattern slice is given, the matcher is set to sourceignore.NewDefaultMatcher. The `GitRepository` now loads the ignore patterns before archiving the repository contents by calling `sourceignore.LoadIgnorePatterns` and other helpers. The loading behavior is **breaking** as `.sourceignore` files in the (subdirectories of the) repository are now still taken into account if `spec.ignore` for a resource is defined, overwriting is still possible by creating an overwriting rule in the `spec.ignore` of the resource. This change also makes it possible for the `BucketReconciler` to not configure a callback at all and prevent looking for ignore matches twice. To finalize the bucket refactor, a change to the reconciler has been made to look for a `.sourceignore` file in the root of the bucket to provide an additional way of configuring (global) exclusions. Signed-off-by: Hidde Beydals <hello@hidde.co>
hiddeco
force-pushed
the
efficient-bucket-download
branch
from
376f72a
to
101814d
Compare
hiddeco
added
area/bucket
squaremo
reviewed
Apr 13, 2021
squaremo
reviewed
Apr 13, 2021
| return func(p string, fi os.FileInfo) bool { | ||
| // The directory is always false as the archiver does already skip | ||
| // directories. | ||
| return matcher.Match(strings.Split(p, string(filepath.Separator)), false) |
There was a problem hiding this comment.
Would https://pkg.go.dev/path/filepath@go1.16.2#SplitList work here?
There was a problem hiding this comment.
I somehow went looking for this method but could not find it :-S. Thanks!
There was a problem hiding this comment.
Or no, it was not. I came across this method as well but it splits them by filepath.ListSeparator, resulting in e.g. [/a/b/c, /d/f/g] instead of the [a, b, c] we are after for the domain.
There was a problem hiding this comment.
Curses! I have made exactly this mistake three or four times now aaaaa
This commit updates Go to 1.16, a required change because of the use of `os.WriteFile` in one of the tests introduced by commit b5004a9. Normally _just_ this would not justify the change, but given the introduction of breaking changes (and thereby forcing a MINOR update anyway), and the various file{system, path} improvements introduced in Go 1.16 like [`filepath#WalkDir`](https://golang.org/pkg/path/filepath/#WalkDir), going ahead with this should be fine. Signed-off-by: Hidde Beydals <hello@hidde.co>
hiddeco
force-pushed
the
efficient-bucket-download
branch
from
101814d
to
d3bcc6a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #333
This PR makes the
BucketReconcilermore efficient by looking forexclusions while downloading files, instead of during the archiving of
the downloaded contents.
It also makes the filtering applied during the archiving
configurable by introducing an optional
ArchiveFileFiltercallback argument and a
SourceIgnoreFilterimplementation.SourceIgnoreFilterfilters out files matchingsourceignore.VCSPatterns and any of the provided patterns.
If an empty gitignore.Pattern slice is given, the matcher is set to
sourceignore.NewDefaultMatcher.
The
GitRepositoryReconcilernow loads the ignore patternsbefore archiving the repository contents by calling
sourceignore.LoadIgnorePatternsand other helpers. The loadingbehavior is breaking as
.sourceignorefiles in the (subdirectories of the)repository are now still taken into account if
spec.ignorefor a resourceis defined, overwriting is still possible by creating an overwriting rule
in the
spec.ignoreof the resource.