build: all golang dependencies seem to be built every time

[jbw976]

Build times seem to have increased recently and it appears that all packages (especially vendored ones) are being built every time. We expected a bit of a longer build because we are using dep ensure to ensure that our dependencies are always up to date, but that may have caused a regression (see #1149).

My local build time seems to have increased to about 2 min every build, even when there are no changes to build.

This is a sample from a build i just ran immediately after a successful build prior to it, with no changes in between:

jared@jared-macbook ~/go/src/github.com/rook/rook
> time make -j4 build
=== ensuring vendor dependencies are up to date
=== go vet
=== go build darwin_amd64
github.com/rook/rook/vendor/github.com/gogo/protobuf/sortkeys
github.com/rook/rook/vendor/github.com/mailru/easyjson/jlexer
github.com/rook/rook/vendor/github.com/gogo/protobuf/proto
<omitted for brevity>
sha256:54b8c643dfd3fb8cbe10a4c9b0034a61893536adc12c77a2362f222665503dbd
=== caching image build-87841a36/toolbox-base-amd64

real    2m0.509s
user    3m56.633s
sys    0m36.498s

Note that all the vendored packages are listed in the build output instead of being skipped.

[bassam]

It looks like dep ensure updates the timestamps on the /vendor directory causing a full rebuild every time:

✔ ~/Projects/src/github.com/rook/rook [master|⚑ 4] 
18:46 $ stat -x vendor/k8s.io/api/core/v1/register.go 
  File: "vendor/k8s.io/api/core/v1/register.go"
  Size: 2761         FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/  bassam)  Gid: (   20/   staff)
Device: 1,4   Inode: 8606611047    Links: 1
Access: Tue Nov  7 18:45:28 2017
Modify: Tue Nov  7 18:42:20 2017
Change: Tue Nov  7 18:42:20 2017
✔ ~/Projects/src/github.com/rook/rook [master|⚑ 4] 
18:46 $ dep ensure
✔ ~/Projects/src/github.com/rook/rook [master|⚑ 4] 
18:46 $ stat -x vendor/k8s.io/api/core/v1/register.go 
  File: "vendor/k8s.io/api/core/v1/register.go"
  Size: 2761         FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/  bassam)  Gid: (   20/   staff)
Device: 1,4   Inode: 8606650895    Links: 1
Access: Tue Nov  7 18:46:29 2017
Modify: Tue Nov  7 18:46:29 2017
Change: Tue Nov  7 18:46:29 2017

[bassam]

this will not get fixed until vendor verification is implemented in dep:

Note: until we ship vendor verification, we can't efficiently perform the Gopkg.lock <-> vendor/ comparison, so dep ensure unconditionally regenerates all of vendor/ to be safe.

also see golang/dep#121

[bassam]

One possibly workaround is to do the following:

.PHONY: go.vendor
go.vendor: $(DEP)
	@if [ ! -d $(GO_VENDOR_DIR) ] || [ ! $(DEP) ensure -no-vendor -dry-run &> /dev/null ]; then \
		echo === updating vendor dependencies ;\
		$(DEP) ensure ;\
	fi

this will only run ensure if the Gopkg.lock file has changed. This will not catch the case where files in the vendor directory have really changed or are non-existent, but these seem like infrequent cases.

[jbw976]

I would love to see any improvement we can make here @bassam to trim down the dev build iteration loop again. Your workaround looks pretty reasonable to me.

[sdboyer]

hi folks - just saw this on a backlink on one of the dep issues. re: @bassam's workaround, #1177 (comment), i just wanted to quickly point you all to a draft plan for a command that will obviate the need for a hack like that.

hopefully we'll be able to get that implemented in the next couple months.