Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/gitmirror: smarter git exporter filtering #11224

Open
bradfitz opened this Issue Jun 15, 2015 · 3 comments

Comments

Projects
None yet
2 participants
@bradfitz
Copy link
Member

bradfitz commented Jun 15, 2015

Add the capability for the watcher's git archive server to optionally filter out content.

Then when the coordinator is asking for the "talks" repo, it can skip the 30+ MB of binaries.

We already do this with the .gitattributes file (see https://golang.org/cl/11050 for #11199) but that's a hack. We should revert that once the coordinator knows how to ask the watcher to filter stuff.

/cc @adg @minux

@bradfitz bradfitz added the Builders label Jun 15, 2015

@bradfitz bradfitz added this to the Unplanned milestone Jun 15, 2015

@bradfitz bradfitz changed the title x/build: smarter git exporter in the watcher x/build/cmd/gitmirror: smarter git exporter in the watcher Feb 11, 2017

@bradfitz bradfitz changed the title x/build/cmd/gitmirror: smarter git exporter in the watcher x/build/cmd/gitmirror: smarter git exporter filtering Feb 17, 2017

@bradfitz bradfitz self-assigned this Feb 17, 2017

@kevinburke

This comment has been minimized.

Copy link
Contributor

kevinburke commented Feb 18, 2017

I can take this one.

@kevinburke

This comment has been minimized.

Copy link
Contributor

kevinburke commented Feb 18, 2017

Thoughts on how the coordinator should ask to exclude files?

GET /build.tar.gz?exclude=foo.jpg&excludeGlob=*.png
@kevinburke

This comment has been minimized.

Copy link
Contributor

kevinburke commented Feb 18, 2017

This might be a little tricky. If I am reading correctly the only way to exclude files from git archive is via a .gitattributes file. We could write this file based on the "ignore" values of the incoming request, but if I am reading correctly git archive hardcodes the .gitattributes filename, so we can only have one incoming request served at a time and that request has to clobber the .gitattributes file - you can't do like git archive --with-attributes=/tmp/foo/.gitattributes.

Another option would be to write the gzip file, open it with a gzip.Reader, and strip files out that we want to exclude before sending it over the wire.

https://git-scm.com/docs/git-archive

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.