-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Smarter glide install procedure #313
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I miss, is that when I now run make sync
explicitly, that it will not check if my vendor folder really has the right content. I tried deleting something in the vendor folder, and then I could not re-sync.
Makefile
Outdated
@@ -32,8 +32,8 @@ distclean: clean | |||
find vendor/ -maxdepth 1 -mindepth 1 -not -name vendor.json -exec rm {} -rf \; | |||
rm -f manifest/*.yaml | |||
|
|||
sync: | |||
glide install | |||
sync: .glide.status |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file needs to be deleted on make distclean
then.
Skipping checking existence of every single file in the vendor/ directory is the very reason for this change - that checking significantly slows down build cycles to check for a problem that is of the developer's own making. If you want to purge vendor code, don't delete individual files in the vendor dir, just blow away the whole tree and let it be recreated from scratch. |
I understand that. It is just counter intuitive, that when I run |
|
In fairness, "glide install --strip-vendor" only adds 5 second clock time to a build (assuming everything is cached). Characterizing that as a significant delay might be overstating the problem a bit. @rmohr is right--we could do better. I added a lazy sanity check--if the vendor folder is newer than .glide.status a full sync needs to occur. This won't cover cases where something deeply nested in the vendor folder has changed, but changes directly inside the vendor folder will be noticed. |
@stu-gott on my machine running "glide install --strip-vendor" adds 15 seconds - the actual compile of go code can be as little as 5 seconds, depending on scope of what has changed since previous compile. That is a significant delay. |
Makefile
Outdated
test -d vendor && \ | ||
test -f .glide.status && \ | ||
test '!' .glide.status -ot vendor || \ | ||
glide install --strip-vendor && touch .glide.status |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, that goes into a way too complex direction for me. Can't we do something simple, like do a .glide.status check when I run "make build", and do an explicit glide install --strip-vendor
, when I run make sync
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We currently call "make sync" twice inside our build process.
EDIT: that is to say--when ./cluster/sync.sh is called
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, so why not doing something like:
build: .glide.status fmt vet compile
.glide.status: glide.yaml
glide update --strip-vendor && touch $@
sync:
glide install --strip-vendor
Then you have a lightweight check when you build code, which should give you a reasonable good dependency update experience, and a make sync
target, which allows you to be sure that everything is up to date.
Makefile
Outdated
@@ -59,4 +63,9 @@ vagrant-sync-build: build | |||
vagrant-deploy: vagrant-sync-config vagrant-sync-build | |||
export KUBECTL="cluster/kubectl.sh --core" && ./cluster/deploy.sh | |||
|
|||
.glide.status: glide.yaml | |||
if test -f $@; then \ | |||
glide update --strip-vendor && touch $@; \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it her glide update
and glide install
on the sync target?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This way glide update only gets triggered if glide.yaml changed. The check for glide install needs to be done every time sync activated. Part of me wants to take this entire section out--and make glide update be something that's only ever done intentionally (and manually).
Since we don't have a unified consensus on how this patch needs to work, I'm going to split the fix for using "glide install --strip-vendor" into a separate PR. |
aa6de2d
to
19b61f5
Compare
Please note that this PR is now rebased onto: #323 |
Makefile
Outdated
@@ -59,4 +63,9 @@ vagrant-sync-build: build | |||
vagrant-deploy: vagrant-sync-config vagrant-sync-build | |||
export KUBECTL="cluster/kubectl.sh --core" && ./cluster/deploy.sh | |||
|
|||
.glide.status: glide.yaml | |||
if test -f $@; then \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are you testing for the existence of the file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Upon first switching to this approach (or after using distclean), all developer's systems will have a glide.yaml but not a .glide.status. It makes more sense to assume an "install" is needed in that case--as opposed to running update--which would get everybody's glide.lock out of sync
Makefile
Outdated
test -d vendor && \ | ||
test -f .glide.status && \ | ||
test '!' .glide.status -ot vendor || \ | ||
glide install --strip-vendor && touch .glide.status |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, so why not doing something like:
build: .glide.status fmt vet compile
.glide.status: glide.yaml
glide update --strip-vendor && touch $@
sync:
glide install --strip-vendor
Then you have a lightweight check when you build code, which should give you a reasonable good dependency update experience, and a make sync
target, which allows you to be sure that everything is up to date.
19b61f5
to
3a186b4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I think, if we want to keep the vendor fetching as part of the build process (which I like, but what we don't have to), then we have to make that a little bit smarter:
First some clarifications:
-
glide update --skip-vendor
works with theglide.yaml
file. It download all dependencies there, goes recursively through the nested dependencies, flattens them, installs them and creates aglide.lock
file. -
glide install --skip-vendor
works with theglide.lock
file. It just goes through the list and installs the mentioned dependencies in that list in the vendor folder. -
Both
glide.yaml
andglide.lock
are checked in to git when they are changed. Therefore, for as long as you don't change dependencies, you only have to runglide install --skip-vendor
, and not a fullglide update --skip-vendor
when you update your git repository.
Then some possible workflows:
- When I change a dependency, I run
make sync
, to update myglide.lock
file, and then I can commit everything.
Here glide update
would always be run.
- When I rebase/checkout the GIT repo, I want to be able to just run
make
, without having to care about dependencies.
To allow the second flow, make like rules which work on timestamps are not good enough. However, we could introduce a .glide.lock.md5
and a .glide.yaml.md5
file. Then, in the corresponding force-make targets, we check if the md5 sum has changed, and only then touch/update .glide.lock.md5
with the new md5 sum, which would then cause a glide install
or glide update
invocation.
This also plays nice with the make sync
invocation, because the md5 sum in the status files only changes, if really one of the files has changed.
Makefile
Outdated
rm -f .glide.status | ||
|
||
checksync: .glide.status | ||
test -d vendor || make sync | ||
|
||
sync: | ||
glide install --strip-vendor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for asking so many questions, but is not glide install
the one which takes only glide.lock into account (with other words the cheaper operation?)? AFAIK glide update
is more expensive, since it goes through glide.yaml, checks out all dependencies, goes through all subdependencies there, prepares the vendor folder, and finally updates the glide.lock
file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is correct. glide install
is cheaper and uses a cache. glide update
crawls all deps and refreshes glide.lock
.
Makefile
Outdated
rm -f .glide.status | ||
|
||
checksync: .glide.status | ||
test -d vendor || make sync |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
glide does not need the vendor folder, why do you check for it? It will create it if it is not there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the current implementation if glide.yaml
is updated, glide update
gets called right before this check. Without this check glide install
will blow away and re-install a freshly laid vendor
folder.
Makefile
Outdated
.PHONY: build fmt test clean distclean sync docker manifests vet publish vagrant-sync-config vagrant-sync-build vagrant-deploy functest | ||
.glide.status: glide.yaml | ||
if test -f $@; then \ | ||
glide update --strip-vendor; \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think glide install
is correct here.
We're pretty much in perfect agreement with regards to your clarifications above. The only difference between what you're envisioning and what's in place currently is what make sync is supposed to do. right now:
However, you make an extremely good point about checking out different branches with possibly different glide.lock files--this
EDIT: It turns out, I do mind if glide install is run every time. When executing |
Signed-off-by: Stu Gott <sgott@redhat.com>
3a186b4
to
528849e
Compare
New approach:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe make sync
can be changed, but good enough for now.
${HASH} glide.yaml > .glide.yaml.hash; \ | ||
${HASH} glide.lock > .glide.lock.hash; \ | ||
elif [ "`${HASH} glide.lock`" != "`cat .glide.lock.hash`" ]; then \ | ||
make sync; \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
glid update seems to be right here
${HASH} glide.lock > .glide.lock.hash; \ | ||
elif [ "`${HASH} glide.lock`" != "`cat .glide.lock.hash`" ]; then \ | ||
make sync; \ | ||
fi | ||
|
||
sync: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would have expected a glide update here
* Do file conversion using go pkgs instead of cli tools Refactored copyIfNotPresent for clarity doTar uses archive/tar instead of cli * Convert xz and gzip to use go pkgs Remove vendor from .gitignore * Exchange github.com/xi2/xz with github.com/ulikunitz/xz dependency for writer functionality Fixed xz import path * Review comments
Also fixes an error: glide install was not stripping vendor folder.
Signed-off-by: Stu Gott sgott@redhat.com