Kind build broken

[costinm]

# sigs.k8s.io/kind/pkg/kustomize
../../sigs.k8s.io/kind/pkg/kustomize/kustomize.go:108:7: undefined: k8sdeps.NewFactory
../../sigs.k8s.io/kind/pkg/kustomize/kustomize.go:109:26: not enough arguments in call to build.NewCmdBuild

Probably caused by removal of vendor.,

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• kind version: (use kind version):
• Kubernetes version: (use kubectl version):
• Docker version: (use docker info):
• OS (e.g. from /etc/os-release):

[BenTheElder]

yes this is caused by the removal of vendor, I'm working on documentation updates, kind requires go modules now, go 1.12+ should be used and modules enabled.

there is now a makefile that will build with a new enough go in docker reproducibly without requiring a particular go version on the host.

[costinm]

With kind modules - it still fails:

go: finding sigs.k8s.io/kind v0.2.1
go: finding github.com/ghodss/yaml v1.0.0
go: finding github.com/go-openapi/jsonpointer v0.17.2
go: finding github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: finding github.com/inconshreveable/mousetrap v1.0.0
go: finding github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742
go: finding github.com/golang/lint v0.0.0-20180702182130-06c8688daad7
go: finding sigs.k8s.io/yaml v1.1.0
go: finding k8s.io/klog v0.1.0
go: finding golang.org/x/lint v0.0.0-20180702182130-06c8688daad7
go: finding gopkg.in/gemnasium/logrus-airbrake-hook.v2 v2.1.2
go: finding k8s.io/client-go v7.0.0+incompatible
go: finding github.com/modern-go/reflect2 v1.0.1
go: finding github.com/go-openapi/jsonpointer v0.19.0
go: finding github.com/emicklei/go-restful v2.8.0+incompatible
go: finding github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf
go: finding github.com/gogo/protobuf v1.1.1
go: finding github.com/golang/lint latest
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding k8s.io/klog v0.3.0
go: finding golang.org/x/lint latest
go: finding gopkg.in/airbrake/gobrake.v2 v2.0.9
go: finding github.com/onsi/gomega v1.4.3
go: finding github.com/modern-go/concurrent latest
go: finding github.com/spf13/pflag v1.0.2
go: finding golang.org/x/tools v0.0.0-20180911133044-677d2ff680c1
go: finding k8s.io/apimachinery v0.0.0-20181126191516-4a9a8137c0a1
go: finding k8s.io/kube-openapi v0.0.0-20181025202442-3a9b63ab1e39
go: finding github.com/google/gofuzz v1.0.0
go: finding github.com/pkg/errors v0.8.0
go: finding github.com/json-iterator/go v1.1.5
go: finding github.com/onsi/gomega v1.5.0
go: finding github.com/spf13/pflag v1.0.3
go: finding github.com/gogo/protobuf v1.2.1
go: finding github.com/pkg/errors v0.8.1
go: finding github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329
go: finding github.com/go-openapi/spec v0.17.2
go: finding k8s.io/kube-openapi latest
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/jteeuwen/go-bindata v0.0.0-20180305030458-6025e8de665b
go: finding k8s.io/apimachinery latest
go: finding github.com/googleapis/gnostic v0.2.0
go: finding github.com/json-iterator/go v1.1.6
go: finding github.com/emicklei/go-restful v2.9.3+incompatible
go: finding golang.org/x/text v0.3.0
go: finding github.com/mailru/easyjson latest
go: finding gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding golang.org/x/tools latest
go: finding github.com/onsi/ginkgo v1.7.0
go: finding github.com/hpcloud/tail v1.0.0
go: finding github.com/go-openapi/spec v0.19.0
go: finding k8s.io/code-generator v0.0.0-20181116211957-405721ab9678
go: finding github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: finding github.com/jteeuwen/go-bindata v3.0.7+incompatible
go: finding github.com/go-openapi/jsonreference v0.17.2
go: finding golang.org/x/net v0.0.0-20190311183353-d8887717615a
go: finding golang.org/x/text v0.3.2
go: finding k8s.io/client-go v11.0.0+incompatible
go: finding github.com/golang/protobuf v1.2.0
go: finding github.com/onsi/ginkgo v1.8.0
go: finding gopkg.in/yaml.v2 v2.2.2
go: finding k8s.io/gengo v0.0.0-20181113154421-fd15ee9cc2f7
go: finding golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e
go: finding gopkg.in/tomb.v1 latest
go: finding golang.org/x/net v0.0.0-20181005035420-146acd28ed58
go: finding github.com/spf13/cobra v0.0.3
go: finding github.com/go-openapi/jsonreference v0.19.0
go: finding k8s.io/code-generator latest
go: finding golang.org/x/crypto v0.0.0-20180910181607-0e37d006457b
go: finding golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: finding golang.org/x/net v0.0.0-20180906233101-161cd47e91fd
go: finding github.com/PuerkitoBio/urlesc latest
go: finding gopkg.in/inf.v0 v0.9.1
go: finding k8s.io/gengo latest
go: finding github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d
go: finding golang.org/x/net latest
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding gopkg.in/inf.v0 v0.9.0
go: finding github.com/onsi/gomega v0.0.0-20190113212917-5533ce8a0da3
go: finding github.com/golang/protobuf v1.3.1
go: finding golang.org/x/sync latest
go: finding github.com/go-openapi/jsonpointer v0.17.0
go: finding golang.org/x/sys v0.0.0-20190312061237-fead79001313
go: finding github.com/google/uuid v1.0.0
go: finding github.com/stretchr/testify v1.2.2
go: finding github.com/spf13/pflag v1.0.1
go: finding github.com/go-openapi/jsonreference v0.17.0
go: finding golang.org/x/crypto latest
go: finding github.com/hashicorp/golang-lru v0.5.0
go: finding golang.org/x/sys latest
go: finding github.com/go-openapi/swag v0.17.0
go: finding github.com/evanphx/json-patch v3.0.0+incompatible
go: finding github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e
go: finding github.com/google/uuid v1.1.1
go: finding github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96
go: finding github.com/go-openapi/swag v0.17.2
go: finding github.com/stretchr/testify v1.3.0
go: finding k8s.io/kube-openapi v0.0.0-20190228160746-b3a7cee44a30
go: finding github.com/PuerkitoBio/purell v1.1.0
go: finding github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903
go: finding github.com/hashicorp/golang-lru v0.5.1
go: finding github.com/go-openapi/swag v0.19.0
go: finding github.com/elazarl/goproxy latest
go: finding github.com/sirupsen/logrus v1.0.6
go: github.com/golang/lint@v0.0.0-20190409202823-959b441ac422: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e
go: finding golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
go: finding github.com/onsi/ginkgo v1.6.0
go: finding golang.org/x/net v0.0.0-20190206173232-65e2d4e15006
go: finding k8s.io/gengo v0.0.0-20190116091435-f8a0810f38af
go: finding gonum.org/v1/netlib v0.0.0-20190331212654-76723241ea4e
go: finding sigs.k8s.io/kustomize v2.0.1+incompatible
go: finding gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485
go: finding github.com/kisielk/errcheck v1.1.0
go: finding github.com/fsnotify/fsnotify v1.4.7
go: finding k8s.io/api v0.0.0-20181026145037-6e4b5aa967ee
go: finding golang.org/x/text v0.3.1-0.20181227161524-e6919f6577db
go: finding github.com/evanphx/json-patch v4.2.0+incompatible
go: finding golang.org/x/tools v0.0.0-20190328211700-ab21143f2384
go: finding github.com/evanphx/json-patch v0.0.0-20190203023257-5858425f7550
go: finding github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f
go: finding github.com/json-iterator/go v0.0.0-20180701071628-ab8a2e0c74be
go: finding github.com/gogo/protobuf v0.0.0-20171007142547-342cbe0a0415
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding gopkg.in/fsnotify.v1 v1.4.7
go get: error loading module requirements
The command '/bin/sh -c go get -u sigs.k8s.io/kind' returned a non-zero code: 1

[costinm]

Can you add back vendor until people get a chance to transition ? Breaking the world is really bad practice.

[BenTheElder]

#510 fixed modules.

Breaking the world is really bad practice.

master is not stable. We can spell this out in the docs, we have release tags for stable versions. Kind is alpha.

[BenTheElder]

can you clone at install from a stable commit, or use a release binary?

https://github.com/kubernetes-sigs/kind/releases/tag/0.2.1

[BenTheElder]

development in master and cut stable releases is standard practice for projects in the kubernetes org, including the main repo.

[mitar]

But there are no instructions how to install "cut stable releases". The only instruction in README installs the master.

[BenTheElder]

that is true, installation can just be download the binary and put it in your $PATH somewhere, but we should document this. I thought we had already, but it turns out we haven't!

binaries are available at EG https://github.com/kubernetes-sigs/kind/releases/download/0.2.1/kind-linux-amd64

altenratively it is git clone https://github.com/kubernetes-sigs/kind && git checkout v0.2.1 && go install .

[BenTheElder]

instructions in the README and docs were updated and tested in clean environments, they should work.

[costinm]

Thanks for updating README - but I don't think 'go mod vendor' is such a huge burden and will allow more people to use or test master and keep 'go get' working.

0.2.1 is from Mar 27 - and it seems Kind is evolving quite rapidly.

We have a similar problem in Istio with master not being stable, so I can't complain much - but at least
we try to keep it compiling. And we have quite a few debates about doing unstable development on
a dev branch, and keeping master always stable and releasable.

[morvencao]

+1 to make master more stable.

[BenTheElder]

Thanks for updating README - but I don't think 'go mod vendor' is such a huge burden and will allow more people to use or test master and keep 'go get' working.

That is true, the burden isn't the concern so much as the git bloat. go get with a module aware go without any options should work once we have another tag. (see below) The ecosystem has moved this way, all supported go versions support this, and we have the makefile for anyone without new enough go.

0.2.1 is from Mar 27 - and it seems Kind is evolving quite rapidly.

Yes, but we have intentionally not tagged a release while we finish making sure the changes are ironed out. Releases should "just work". The next one will be between now and KubeCon (end of next week) if we can finish ensuring the changes are good to go.

We aim for ~monthly releases currently. We're slightly overdue due to trying to finish ironing out the breaking image improvements. It should work much faster,

We have a similar problem in Istio with master not being stable, so I can't complain much - but at least we try to keep it compiling. And we have quite a few debates about doing unstable development on a dev branch, and keeping master always stable and releasable.

I will again point out that master did and always has compiled, and in fact passed conformance against Kubernetes 1.12, 1.13, 1.14, and master (and until recently 1.11, but kubernetes stopped supporting it) we have pretty thorough CI. All PRs must pass this.

Master is "unstable" in that we don't have the resources to fully guarantee it. Releases get a little extra QA. EG we do not have access to Windows CI, but we test there before releasing. Most of this is just me :-)

We did however break the node image format between releases, and we changed some implementation details of the node. We make no guarantees about this yet, kind is alpha. We're pushing to figure out what changes to make on the way to beta and GA/stable.

[costinm]

The second report

...
go: finding github.com/sirupsen/logrus v1.0.6
go: github.com/golang/lint@v0.0.0-20190409202823-959b441ac422: parsing go.mod: unexpected module path "golang.org/x/lint"
...

was from a 'go get ' with modules enabled.

We are also switching to go mod - but some form of vendor is still useful. Disk is not that expensive. There are few other options - like having a second git repo with just a main and vendor. Fetching the deps from a single git (packed/compressed/etc) is faster than fetching from large number of git repos,
and the mod cache is tricky to setup.

[BenTheElder]

See the updated readme, the @master part is breifly required until we put 0.3 out, otherwise it tries to get 0.2.1 which has github.com/golang/lint instead of the golang.org/x/lint import.

I'm also looking at making tools a second module.

There are few other options - like having a second git repo with just a main and vendor. Fetching the deps from a single git (packed/compressed/etc) is faster than fetching from large number of git repos,
and the mod cache is tricky to setup.

Getting a second repo is wildly painful in this project and unlikely to be approved for that purpose.

As for speed, https://proxy.golang.org/ pretty much solved that issue.

[neolit123]

@ all
pinning your CI against the tip of the kind master branch is far from recommended...
i encourage you to pin against a fixed SHA that works for you and updating this SHA periodically in case you want new kind features.

we do the same mistake in the kubernetes CI jobs, but at least we know when and how we are going to break it.

[neolit123]

@BenTheElder

See the updated readme, the @master part is breifly required until we put 0.3 out, otherwise it tries to get 0.2.1 which has github.com/golang/lint instead of the golang.org/x/lint import.

might be a good idea to not depend on the lint package?
unless the verify tools demand air-gaped support i don't see that much overhead in pulling the package and rebuilding it on demand outside of the scope of /go.mod

EDIT: i guess the followup on that is #514

[BenTheElder]

Agreed @neolit123, see: #514 It's just to pin for CI. #514 moves all dev time tools to their own module. That way we also don't require all the code-generator code when just building kind. *From: *Lubomir I. Ivanov <notifications@github.com> *Date: *Thu, May 9, 2019, 08:55 *To: *kubernetes-sigs/kind *Cc: *Benjamin Elder, Mention @BenTheElder <https://github.com/BenTheElder>

…

See the updated readme, the @master <https://github.com/master> part is breifly required until we put 0.3 out, otherwise it tries to get 0.2.1 which has github.com/golang/lint instead of the golang.org/x/lint import. might be a good idea to not depend on the lint package? unless the verify tools demand air-gaped support i don't see that much overhead in pulling the package and rebuilding it on demand outside of the scope of /go.mod — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#509 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHADK4WBUXLWQWQV6ZXCPDPURCOVANCNFSM4HLWDN5Q> .

[costinm]

I think ideally our 'daily' build should use a daily build of Kind, our weekly builds should use weekly builds of kind, and same for releases. On the short term - we have the problem that 'master' is far better than the last release ( speed, fixes, etc).

Regarding 'build tools' - as I mentioned in a different bug, it would be ideal if Kind publishes docker
images with kind binary included, ready to use ( maybe with cached images to speed up creation of cluster), and ideally variants usable in major CI/CD systems ( like CircleCI, BuildKite, Prow, Gitlab, Concourse, Drone, GCB) .
We found that having hermetic docker images with all the tooling needed to build a project is
both speeding up the build and help reproduce builds in future. We are planning to include kind
binaries in our (almost) hermetic build tool image, so it won't be built from master.

[BenTheElder]

Please consider using a specific commit / release for all of your builds. we can't know what will and won't break your CI and we need to iterate to build out the features everyone is demanding and improve kind.

Regarding 'build tools' - as I mentioned in a different bug, it would be ideal if Kind publishes docker
images with kind binary included, ready to use ( maybe with cached images to speed up creation of cluster), and ideally variants usable in major CI/CD systems ( like CircleCI, BuildKite, Prow, Gitlab, Concourse, Drone, GCB) .

That sounds nice ... but

• We really shouldn't be publishing more images to the dockerhub, that's supposed to be an unnoficial-ish stopgap while kubernetes sorts out how subprojects publish images. there is process to ensure resources are community owned, and we had to sidestep this temporarily to bootstrap the project.
• We really don't have resources to actively tackle that many CIs currently. I'd be happy to help ensure kind works in one of those for istio, but right now taking on all of those is a bit much

FWIW, the kind node image has all of the dependent images cached in it hermetically. Caching that image as appropriate for your CI setup will make it work without the internet.

We found that having hermetic docker images with all the tooling needed to build a project is
both speeding up the build and help reproduce builds in future. We are planning to include kind
binaries in our (almost) hermetic build tool image, so it won't be built from master.

kind create cluster --image=foo@sha256:bar should be reproducible. at HEAD we now have a reproducible build (make) we'll use for the binaries going forward, a CI setup can download those binaries. they're small.

As I said above, the cluster creation itself should be about as reproducible as possible, the kubernetes setup is hermetic. sticking this hermetic setup into another one doesn't really make it any more hermetic, just pin which kind binary and image you use. (pin to a commit, pin to a node image SHA256)

[costinm]

I think having at least one pipeline using master is important - it doesn't have to be pre-submit or blocking, but it is good to catch problems earlier and provide feedback.
We are now taking snapshots and have them in our hermetic build tool images.

Re. CIs to tackle - you don't have to take all of them, but it would be great if you can get GCB working.
For the others - I assume community will provide examples. For Istio, we had community members get
Kind working in Concourse - and we also got it working in Circle (machine/remote_docker) and BuildKite(vm) - happy to continue to test Kind in those CIs. I am sure community members using other
CI/CDs will also contribute fixes or scripts to get Kind working there.

Re. image cached in the node - that's what we want to do for Istio as well, have 'air-gapped' images
that include everything needed to build/run/test Istio and Kind without any network access.

On publishing: if you just add Kind binary to the node image you already publish, it would be enough. Users can then either extract from that node image or use it with a different entrypoint.
We are extending the node image you publish to add Istio tools and images. Doesn't matter where you
publish it - I'm sure there will be a place. Happy to publish kind images in Istio dockerhub (with Istio added on top) :-) - the goal is to have a very easy and fast way to develop/test.

I noticed that different versions of Kind binary don't work with different node images - would be good to
clarify what are the expectation. I.e. Kind binary v1 builds a node image image1, kind binary v2 builds a node image2 - will kind v1 be able to use image2 ? Will kind v2 use image1 ? What is the relation between different kind versions and images built with 'kind build' of different versions ?
If each kind version should use with nodes images build with its own version - we need to keep them in sync.

[alexellis]

cc @LucasRoesler

This also hit us and affects all our CI and documentation 😞

The initial fix for one project meant doing this: openfaas/ofc-bootstrap@50410e0

I'd really appreciate seeing the following improved upon: #208 as I hear the latest binary is from March.

[BenTheElder]

I think having at least one pipeline using master is important - it doesn't have to be pre-submit or blocking, but it is good to catch problems earlier and provide feedback.

Again, we have extensive presubmit and CI testing on prow. We know it works and passes conformance for 1.12..master

Re. CIs to tackle - you don't have to take all of them, but it would be great if you can get GCB working.
For the others - I assume community will provide examples. For Istio, we had community members get
Kind working in Concourse - and we also got it working in Circle (machine/remote_docker) and BuildKite(vm) - happy to continue to test Kind in those CIs. I am sure community members using other
CI/CDs will also contribute fixes or scripts to get Kind working there.

@munnerz span up a project for this, it's a work in progress, started with circleCI in absence of a particular request.

On publishing: if you just add Kind binary to the node image you already publish, it would be enough. Users can then either extract from that node image or use it with a different entrypoint.
We are extending the node image you publish to add Istio tools and images. Doesn't matter where you
publish it - I'm sure there will be a place. Happy to publish kind images in Istio dockerhub (with Istio added on top) :-) - the goal is to have a very easy and fast way to develop/test.

We won't be adding the kind binary to the node image. The node image also does not contain docker for a while now. The node image is a snapshot of Kubernetes and what we need to boot it.

I will consider another image in the future, in the meantime is wget https://github.com/kubernetes-sigs/kind/releases/download/0.2.1/kind-linux-amd64 in your own image a big deal...?

I noticed that different versions of Kind binary don't work with different node images - would be good to
clarify what are the expectation. I.e. Kind binary v1 builds a node image image1, kind binary v2 builds a node image2 - will kind v1 be able to use image2 ? Will kind v2 use image1 ? What is the relation between different kind versions and images built with 'kind build' of different versions ?
If each kind version should use with nodes images build with its own version - we need to keep them in sync.

So until some changes in HEAD that are unreleased all images worked with all releases. the kindest/node:v1.14.1 image requires HEAD and will be in the next release, which will prominently feature a note about the image changes in the release notes. I don't expect to need to change the format again soon, and the plan is to document and not push over any of the old images with the old format, whereas newer images will require 0.3+

We're also going to resume pinning the default image to a sha256, and recommend consumers do the same.

Avoiding breakage is a key tenant, but we are in alpha. I spent multiple weeks trying to get the improvements in without any breakage, but we needed to move forward. It is still only v0.2 after all, we haven't been around all that long!

For 0.4 I'm also aiming to make it easier / cheaper to build node images by supporting building from tarballs, there's an existing PR to fix up, this will make it relatively cheap and easy to pick a kubernetes version and publish an image for it to pin for your project(s) until kind is fully stable.

This also hit us and affects all our CI and documentation 😞

Apologies for that but, please do not consume kind at HEAD in your CI. At least pick a commit periodically and pin that. There are many things that may still need major changes on the way to beta... Networking details are the next big one.

I'd really appreciate seeing the following improved upon: #208 as I hear the latest binary is from March.

Yeah, the cadence currently is about 1.5 months instead of 1 month. This is still twice as fast as Kubernetes and faster than most deployers. I'd like to bring it down to a month, but this release is biting the bullet on some major breaking changes we'd been discussing for a while. As a result the image size and boot times will both drop, and booting should be more reliable.

[BenTheElder]

See https://github.com/kubernetes-sigs/kind/milestone/5 for 0.3, slated by next weekend.

[BenTheElder]

Even ignoring changes like go modules, using kind at HEAD for important CI is a risky bet. Even an innocuous change on our end could somehow break your setup, and kind at HEAD has no SLA. I don't think kops, minikube etc. do either. Even Kubernetes testing has or will be moving to pin a particular kind version.

[alexellis]

My point is that the recommended instructions changed overnight as a breaking change.

From:

go get -u sigs.k8s.io/kind@master && kind create cluster

To:

GO111MODULE="on" go get -u sigs.k8s.io/kind@master && kind create cluster

They should really not change that drastically without some proper warning. It reminds me of Logrus/logrus.

If the maintainers group don't want people to use HEAD (read typing in: go get) then I would like to offer up a suggestion to change the instructions to reflect that. Can that even be done with go get?

[BenTheElder]

My point is that the recommended instructions changed overnight as a breaking change.

That is true, though really we should update to reflect that those are not recommended for CI. It's easy to try out kind that way (also note: with no options / config / flags), but none of the projects I work with use any_ tools from HEAD like this in CI, all of them pin a version to avoid this.

They should really not change that drastically without some proper warning. It reminds me of Logrus/logrus.

Apologies, FWIW we did pin a warning in the #kind slack channel. The intention was for release notes to improve this. Again, I didn't expect CI to actually be installing from HEAD, which was clearly an oversight. The documentation does cover release binaries and I expected that CI would of course use that for stability on such a young project... Clearly this was an oversight, but it is still recommended.

If the maintainers group don't want people to use HEAD (read typing in: go get) then I would like to offer up a suggestion to change the instructions to reflect that. Can that even be done with go get?

Yes, when v0.3.0 is cut in the next week, updating the README to drop the @master will cause go get -u to pick the latest tag by default. [1]
@master obtains the latest commit in the branch, which is a stopgap due to #510 being in v0.2.1.

This is one of the behaviors using modules improves.

[1]: https://github.com/golang/go/wiki/Modules#how-to-upgrade-and-downgrade-dependencies

[BenTheElder]

v0.3.0 is out with binaries now. go get will default to that version, and we've updated all the installation docs to prefer this. Closing this and moving forward with modules now. Also noting that minikube went from go dep to modules and removed vendor in one PR in the past week as well. kubernetes/minikube#4241