Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: should default to use mod=vendor flag if the vendor folder exists #27227

Open
JeremyLoy opened this Issue Aug 25, 2018 · 65 comments

Comments

Projects
None yet
@JeremyLoy
Copy link

JeremyLoy commented Aug 25, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"

What did you do?

  • Given a fresh install of go, there are no files downloaded in $GOPATH/pkg/mod.
  • I cloned a repo that uses the new Go 1.11 modules with vendored dependencies
  • I executed go run main.go to run the application.

What did you expect to see?

  • the application ran without downloading external dependencies.

What did you see instead?

  • The go tool downloaded all dependencies to $GOPATH/pkg/mod, despite the fact that the dependencies were vendored

I know this can easily be fixed by instead go run -mod=vendor main.go, but because the go tool is now being advertised as "context aware", I think this would be a good QoL change.

@bradfitz bradfitz changed the title pkg/mod: go tool should default to use mod=vendor flag if the vendor folder exists cmd/go: should default to use mod=vendor flag if the vendor folder exists Aug 25, 2018

@bradfitz bradfitz added the modules label Aug 25, 2018

@bradfitz bradfitz added this to the Go1.12 milestone Aug 25, 2018

@thepudds

This comment has been minimized.

Copy link

thepudds commented Aug 26, 2018

@JeremyLoy note that while the default is to ignore the vendor directory, you can relatively easily change that behavior with an environmental setting GOFLAGS=-mod=vendor. You could use that in CI, or could set it in a .bashrc or similar for day-to-day use if you want.

This is a new environmental variable in Go 1.11, where one of the justifications for introducing it was specifically to make it easier to opt in to vendoring given some people would want that regularly (e.g., see #26585 or https://go-review.googlesource.com/c/go/+/126656).

From the doc:

GOFLAGS
	A space-separated list of -flag=value settings to apply
	to go commands by default, when the given flag is known by
	the current command. Flags listed on the command-line
	are applied after this list and therefore override it.

One other quick note is that those downloads end up in a shared cached (so even if you don't opt-in to vendoring, it cuts down on how many times you would end up downloading dependencies).

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Aug 26, 2018

I strongly agree with this proposal. If a project committed a vendor folder, this is a strong indicator that vendoring should be used by default. I can’t think of a workflow in which a developer would commit a vendor folder but prefer other developers to ignore it by default.

In other words, if a project decides to use vendoring, then the whole go tool should default to use vendor first, and download/cache only if not there.

@FiloSottile

This comment has been minimized.

Copy link
Member

FiloSottile commented Aug 30, 2018

By the way, this is part of what GOFLAGS was added for: https://go-review.googlesource.com/c/go/+/126656

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Aug 31, 2018

GOFLAGS isn’t the right solution here. This is not a per-user / per-computer decision like “add debug flags”. This is a per-project decision that should affect all users building it. Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.

@JeremyLoy

This comment has been minimized.

Copy link
Author

JeremyLoy commented Sep 2, 2018

WhileGOFLAGS is a decent immediate workaround, I do want to stress the importance of the vendor folder as a part of this issue.

To put it bluntly, Github is not an artifact repository. At any given time, a repository owner can either

  1. delete their account
  2. delete a repository
  3. force push deletions to master
  4. remove a tag
  5. change a tag

To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.

There seems to be a rush to remove vendor from the go toolchain. It is still referenced as an "experiment" and there was a bit of swirl in even getting modules to support vendoring in its first release. Until Project Athens or equivalent becomes mainstream, vendoring should be fully supported (but never required).

EDIT: coincidentally, I found out the official Heroku build pack for Go uses this very workaround. It seems like this really should be the default case.

@myitcv

This comment has been minimized.

Copy link
Member

myitcv commented Sep 3, 2018

@rasky

Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.

But we're talking about working in module mode here, correct? Or by "currently" do you mean GOPATH-mode? Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.

@JeremyLoy

To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.

There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.

As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.

There seems to be a rush to remove vendor from the go toolchain

I don't think this is quite accurate. In GOPATH mode (the default), vendoring works exactly as it has done up until now. In module mode (the opt in) the behaviour changes.

So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Sep 3, 2018

Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.

Yes, I'm speaking about module mode. The point is that "not being the default" for projects that uses vendoring (that is, projects in which developers went through the process of calling go mod vendor and commit the vendor directory) is the wrong default. If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used. People that don't use vendoring don't need to have vendoring explicitly disabled; they simply won't have a vendor directory in their project. By switching the vendoring support off in module mode, go 1.11 is making using vendoring more difficult for no reasons I can understand, without making things simpler for those who don't need or want vendoring.

There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.

Thanks, but that is largely suboptimal. It requires a custom installation step to be repeated for each user, and also handle and commit binary files into the SCM which are harder to handle (especially with git) and review.

As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.

I fail to see how Athens solves the needs of people using vendoring. AFAICT Athens helps with the issue that people can remove/sell GitHub repositories to other parties, and this can break builds and/or even cause security problems. This is not what vendoring is after. Vendoring is required in workflows that need 100% of the source code for a project committed under a single SCM, so that it can be fully audited, versioned, and built completely offline, without any dependency on any third-party service or installation. I deliver code to enterprises that want the code to be complete, not depend on third party services, and fully versioned with their existing code tracking tools. Vendoring solves this problem; Athens doesn't seem to.

So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.

You seem against adding a feature for people that need it for the sake of orthogonality. If you don't want or need vendoring, there's nothing in vendoring support that creates problems to you. Just use modules, your download cache, Athens or whatever you prefer. You will never have a vendor directory in your repos and nothing will break or be suboptimal for you. But if I need vendoring support and commit a vendor directory, why should vendoring support be disabled by default for me, causing more difficult builds for all involved developers?

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Sep 11, 2018

If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.

That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

Giving their users an option is not at all the same thing as providing an explicit recommendation.

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Sep 11, 2018

If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.

That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

I'm not sure that we can deduce anything from what Go 1.11. It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures. What we ended up with is basically non working for many vendoring use cases, hence this bug (and others).

Rather than discussing hypothesis made by people that don't vendor, I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report. I for sure know this doesn't work for my use case as already explained in mine, and others have already chimed in.

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

That's perfectly fine for me at least. In fact, in my use case, vendoring support could go as far as completely subsuming the module download cache and/or go.sum. That is, it would be great that once you opt-in to "vendoring mode", all new modules are automatically expanded into the vendor directory, used by default, removed when they're not used, etc.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Sep 11, 2018

I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report.

That's kind of my point, though: you're requesting feedback from one particular subset (package maintainers that choose to use vendoring) and ignoring another (package consumers, who have no choice in the matter in GOPATH mode, but do now have a choice in module mode).

We need experience reports from everyone — including package maintainers and consumers, and including both open-source and enterprise codebases.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Sep 11, 2018

@JeremyLoy, could you give some more information about your use-case?

The go tool downloaded all dependencies to $GOPATH/pkg/mod, despite the fact that the dependencies were vendored

  • Did you run go mod download after cloning the repository? Why or why not?
  • What is the impact of downloading the dependencies?
    • Do you intentionally want to bypass go.sum verification, or do you have in mind that the vendored dependencies remain unmodified?

Until [module proxies become] mainstream, vendoring should be fully supported (but never required).

Vendoring is still supported in Go 1.11, both in GOPATH mode and with -mod=vendor.
I am not aware of any proposal to remove it in 1.12, and would be surprised to see one at this point.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Sep 11, 2018

@rasky

It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures.

Please remember that among our gopher values is to “be charitable”.

The approach to vendoring in Go 1.11 does seem to have evolved with the implementation, but that doesn't mean it was “retrofitted”. A flag very similar to -mod=vendor was floated in #19234, and the ability to disable vendoring was discussed in #16562; as far as I am aware, both of those significantly predate the current module design.

@JeremyLoy

This comment has been minimized.

Copy link
Author

JeremyLoy commented Sep 11, 2018

@bcmills I'm currently using Go for enterprise application development, so I can give an experience report for that. You'll have to find someone else for the package maintainer report.

We currently vendor our dependencies using dep. We looked into converting to go modules, but found that the differences were too great to warrant a change right now, especially as modules are still in development.

We vendor for a variety of reasons, many of which were already stated here, but I will reiterate them for clarity.

  • a single deliverable - the client is able to take the codebase and build the application from source without internet access.
  • guaranteed stability - the leftpad problem simply cannot happen with vendoring. It is still an issue with both GOPATH and module development
  • security audits - the client is able to audit not just our source code, but our dependencies in a reproducible manner.

we also vendor tools such as gomock, goimports, swagger etc.

Now to answer your questions:

Did you run go mod download after cloning the repository? Why or why not?

no. because with the vendor folder, I should not have to.

Do you intentionally want to bypass go.sum verification

no, quite the opposite in fact. I imagined that go.sum would be a drop-in replacement for dep's Gopkg.lock

do you have in mind that the vendored dependencies remain unmodified?

yes.

Having to set an environment variable or constantly add a flag is a bit of a nuisance, but not a game breaker. The biggest issue to me is that I expected go modules + vendoring to behave exactly like dep with vendoring (algorithmic changes not included)

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Sep 11, 2018

It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures.

Please remember that among our gopher values is to “be charitable”.
The approach to vendoring in Go 1.11 does seem to have evolved with the implementation, but that doesn't mean it was “retrofitted”.

I'm sorry if it was borderline harsh, but anything related to "proposals" in Go often leads to a very frustrating experience, so much that I tend to steer away as much as possible from it and focus on where the community experience is vastly superior (eg: compiler development).

This said, I still think the quoted sentence is factually true. It was clearly stated in Russ' first blogpost:

This proposal keeps the best parts of go get, adds reproducible builds, adopts semantic versioning, eliminates vendoring, deprecates GOPATH in favor of a project-based workflow, and provides for a smooth migration from dep and its predecessors.

and has been repeated other times elsewhere. The vgo design didn't include vendoring; at some point, it was re-added to the picture, but there wasn't any discussion or attempt (at least that I could find) to rethink vendoring in the Go modules world. The fact that a default that doesn't work for people using vendoring was chosen, or that there are large overlaps between the vendor directory and the download cache is probably a consequence of not having fitted vendoring in the picture from day one.

@thepudds

This comment has been minimized.

Copy link

thepudds commented Sep 11, 2018

@rasky I think you are right that the initial set of vgo blog posts initially proposed dropping support for vendoring.

However, there was then a fair amount of feedback from the community about the importance of vendoring. As far as I could see, that community feedback was clearly heard by the core Go team, which then resulted in the proposed plan being updated to actually retain support for vendoring with modules. The feedback was discussed in multiple venues, but this might have been the longest thread:

https://groups.google.com/d/msg/golang-dev/FTMScX1fsYk/uEUSjBAHAwAJ

...where within a week or so of the initial vgo blog post, the core Go team had switched to proposing retaining vendoring support with vgo/modules based on that community feedback.

In any event, sorry if my historical comments are not very interesting. A much more interesting question of course is what should the behavior be in 1.12+.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Sep 12, 2018

The vgo design didn't include vendoring; at some point, it was re-added to the picture, but there wasn't any discussion or attempt (at least that I could find) to rethink vendoring in the Go modules world.

Yeah, I can see that. Bear in mind that the module support in Go 1.11 is still preliminary: we know that it has some rough edges, and vendoring is one of them.

I think it's important that we figure out a path forward, but it's not clear to me whether that will be to improve vendoring and better integrate it into module mode, or to improve the available module operations and deprecate vendoring entirely. I expect we'll try something in at least one of those directions for 1.12, but it's important that we have enough time to try out the improvements and make sure they work for those use-cases before they become the default.

In particular, because vendoring support in module mode is pretty awkward in 1.11, I think it's important not to try to push it as a default: I don't think it's the right design point yet, and I don't want folks to start relying on a design that really needs more thought.

@ondrej-fabry

This comment has been minimized.

Copy link

ondrej-fabry commented Sep 12, 2018

  • a single deliverable - the client is able to take the codebase and build the application from source without internet access.
  • guaranteed stability - the leftpad problem simply cannot happen with vendoring. It is still an issue with both GOPATH and module development
  • security audits - the client is able to audit not just our source code, but our dependencies in a reproducible manner.

@JeremyLoy has made some great points here why vendoring is really needed in the software we build.

Without vendoring how are you planning to solve these points? Force everyone to host some caching/proxy thingy that would serve the dependencies transparently?
This would most likely compel developers to create various different solutions just like with vendor did before (Godeps, govendor..) and thus split the community once again while still not solve all of the points above.

What if some of the dependencies disappears and you don't have it in the proxy cache or anywhere else? This would force developers to fix/replace the dependency in their repository. However what if this is related to multiple versions with ongoing support? It would have to be fixed for all of them..

I'm really curious about your future solutions without the vendoring support that would satisfy the points above while at the same time not force us to manage extra things outside of the repository to successfully build our Go programs.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Sep 12, 2018

@ondrej-fabry

Without vendoring how are you planning to solve these points?

That is a matter for a separate issue (perhaps #27618, and perhaps others as well). If you have specific requirements, please contribute an experience report describing both your use-case and how you address it today. (As I've said, it's not clear to me whether the solution will be to improve vendoring or to replace it, but experience reports will help either way.)

The matter at hand for this issue is whether -mod=vendor should be the default, in module mode, when the module's root directory has a vendor subdirectory.

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Sep 12, 2018

Is there a single experience report, email or comment on GitHub issue that describes a real-world workflow in which the current default is correct? I have not seen any.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Sep 12, 2018

@rasky I don't think we have any experience reports for vendoring with module mode at all. As I've said repeatedly, I think it needs work before we can recommend any particular workflow. For now, if the module support in 1.11 doesn't work for you, don't use it. GOPATH mode is still supported and shouldn't get in your way.

If you want to influence the future direction of module mode, you need to share your actual use-case instead of just arguing that what we have today doesn't work: your current criticism is not constructive. We already know that what we have today doesn't work for you: we need to know how to make that better.

@JeremyLoy

This comment has been minimized.

Copy link
Author

JeremyLoy commented Sep 12, 2018

@bcmills I added a link to my comments (and by proxy, this issue) to the experience report page you referenced.

If you want me to modify the experience report/submit it in a different way etc just let me know!

@driusan

This comment has been minimized.

Copy link

driusan commented Oct 3, 2018

My experience with the current vendor behaviour has been as follows:

I've written a (partial) drop in replacement for git in Go, primarily so that I can use it on Plan 9 where there's no access to the real git client. A user requested that I vendor my dependencies so that it can be bootstrapped without @0intro's rc script hack, which seemed like a simple enough enough request so I decided to oblige.

I copied the files to the vendor directory as I did to vendor with previous versions of Go and it didn't work. I ran "go mod vendor" which appeared to generate some ancilliary files for Go 1.11 like vendor/modules.txt. It appeared to work (I was inside my $GOPATH), so I committed and pushed the results. The same user then filed a GitHub issue saying that it was still trying to pull the code that was supposed to be vendored code from bitbucket instead of using the vendored library when doing go build -x. After some investigation, I found the -mod vendor flag and it resolved his problems. (Most of this took place at driusan/dgit#142)

I support this proposal because the current Go 1.11 behaviour:

  1. Isn't how previous versions of Go worked.
  2. Isn't what package maintainers expect.
  3. Isn't what users expect.
  4. Makes it difficult to build offline, even when all dependencies are vendored.

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Nov 15, 2018

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Nov 15, 2018

I think mod=vendor is too coarse: ideally the vendor folder should function more like an implicit set of replace directives, and we should automatically use whatever modules are found there (and fetch the remaining modules as usual).

That's certainly not happening for 1.12, but it's still on my radar.

@cespare

This comment has been minimized.

Copy link
Contributor

cespare commented Jan 8, 2019

@ondrej-fabry my feedback is specific this particular issue; it is not a report on my experience writing Go.

@theckman

This comment has been minimized.

Copy link
Contributor

theckman commented Jan 30, 2019

It seems that this issue is still marked as "NeedsDecision". The feedback in this ticket has been overwhelmingly in favor of making this change to be the default. The only dissent I've seen here has come from @bcmills @myitcv. I'm also not seeing anyone indicating agreement with their dissent (either via comment or by virtue of emoji).

@bcmills @rsc what else is needed to make a final decision for what will be better for the entire Go community? How can I help us work towards making a final decision on this issue?

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Jan 30, 2019

The issue is already milestoned as 1.13; even if it was implemented today, it wouldn't ship to most users for another six months. Is there a particular urgency to getting this done?

@theckman

This comment has been minimized.

Copy link
Contributor

theckman commented Jan 30, 2019

@mvdan everyone adopting modules now who wants to keep vendoring is going to need to add this extra bit to their build scripts, after spending the time trying to figure out why they keep downloading the world, to then remove it later assuming the default is changed. Seems like it could become a pretty large distributed time investment, and would be a bit of churn.

Edit: I've seen it come up on Slack quite a few times, and this ticket has been open since August.

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Jan 30, 2019

Sure. But hurrying to approve and implement this issue this week is not going to affect users any faster. I personally think that endorsing vendoring as it exists today isn't a great idea for modules, so I'd like to see what approaches arise during the 1.13 cycle. In my opinion, getting the correct solution is better than getting a solution fast.

this ticket has been open since August.

Note that there are likely hundreds of modules-related issues at any point nowadays; there's tons of stuff that has been pushed back to 1.13, and I'd argue this is not the most urgent issue of them all. This is precisely why GO111MODULE=on is not a default yet.

@thepudds

This comment has been minimized.

Copy link

thepudds commented Jan 30, 2019

As far as I was able to follow, it didn’t seem to me that the core go team was against enabling vendoring by default, but rather there was a bit of a question as to exactly how it should behave when enabled by default, and then of course the time needed to implement.

See this recent comment from @bcmills, for example, which suggests what seems to me to be an improvement to behavior (compared to the current 1.11 mod=vendor behavior):

#27227 (comment)

But of course, different people might have different perspectives as to whether not that is an improvement, which seems like a valid thing to discuss or capture in this issue.

@theckman

This comment has been minimized.

Copy link
Contributor

theckman commented Jan 30, 2019

@mvdan I'm not trying to approve and implement this issue this week. I'm looking for a final decision to be made so that the work can start for the Go 1.13 cycle. I'd personally prefer this gets in to 1.12, to short-cut some of the churn, but I don't see that as being a reasonable ask at this point.

I personally think that endorsing vendoring as it exists today isn't a great idea for modules

Can you elaborate further? Why are 100% reproducible builds a bad idea for modules?

I've been on the unlucky end of total loss of an artifact server, and a dependency having been killed in the process. That code was lost and the project could not be compiled. In that case Modules wouldn't have saved me, but vendor would have.

I don't think we should treat vendor as a bunch of replace statements, but instead a way for those who want absolutely reproducible builds to get them. It's my belief it should be a dependency cache that travels with the source repository.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 30, 2019

As far as I can tell we're in agreement that we should use vendor more aggressively by default.

However, there are still some details to decide. For example:

  • Should we substitute modules, individual packages, or the entire transitive source tree? (Probably modules, but then users will have to be careful to keep modules.txt up-to-date. The vendor directory in the pre-modules world applies to individual packages; on the other hand, -mod=vendor today applies to the entire source tree.)
  • Should we modify go mod vendor to support more targeted vendoring? (Perhaps, because that may make it easier to maintain an accurate modules.txt.)

FWIW, I'm planning to address this issue in Go 1.13.

@nomad-software

This comment has been minimized.

Copy link

nomad-software commented Jan 30, 2019

@bcmills I think we need to first keep in mind what problem the vendor folder actually solves. In my mind, it supplies one hundred percent reproducible builds, which don't need an internet connection and protects you from online repositories being removed. That's probably as sophisticated as we need to get.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 30, 2019

@nomad-software, the vendor folder is not necessary at all for 100% reproducible builds in module mode. You can save the contents of the module cache, and use the saved cache as a GOPROXY. (This was all covered in the preceding discussion.)

The vendor directory does not solve reproducibility; in fact, quite the opposite, since you can make arbitrary changes in the vendor contents that will not be reflected in builds initiated from outside your module. (See #29058 and #27348.)

What the vendor directory does provide is a more ergonomic presentation of the cached code: easier auditing without modifying the repo, simpler diffs for version control tools, etc. Some folks may want that for their entire transitive closure of dependencies, but others likely will not.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 30, 2019

@leitzler points out that using vendor by default would change the source code in use for existing modules that have a vendor directory, which actually works against reproducibility for those modules.

That's true, and one of the details I'm contemplating is “when to activate vendor-by-default”: now that go mod tidy adds a go line to the go.mod file, we could trigger the behavior (or not) based on the declared version. For example, perhaps we should only use the vendor directory by default if the go.mod file specifies go 1.13 or later.

@theckman

This comment has been minimized.

Copy link
Contributor

theckman commented Jan 30, 2019

the vendor folder is not necessary at all for 100% reproducible builds in module mode. You can save the contents of the module cache, and use the saved cache as a GOPROXY. (This was all covered in the preceding discussion.)

@bcmills This just sounds like more manual vendoring, where you'll now need additional scripts to overlay the module cache and to resolve / save any new additions. Or you now need to have some sort of process to continually check in your module cache to source control. How do you then use that on different developer workstations and build systems? What tooling will go mod provide to help make that easier?

Thinking about it here it seems complex, with potential for pain in the future. It makes me even more convinced that we should keep the more simple vendor concept as the default.

The vendor directory does not solve reproducibility; in fact, quite the opposite, since you can make arbitrary changes in the vendor contents that will not be reflected in builds initiated from outside your module.

If you assume most people won't be just changing stuff in vendor willy-nilly, it doesn't have the opposite effect. It'd especially be true go mod toolkit were to control that folder and overwrite any changes that are made, and to fail of there were any modifications. I never would advocate for mutating vendor, and if you wanna change something there you need to fork it and have your dependency manager drop the fork in to vendor at the right path. I think replace would handle this nicely.

I also assume that vendor will not be considered for things consuming that module, and that generally the only thing using vendor are tests and the build for package main. We don't run tests from external packages, and we can't import their package main, so that seems fine. And with MVS, new consumers should get the same package versions as in their vendor folder, or the version semantically closest to it.

Now, if one of those vendored dependencies disappears, new consumers will fail to build. But the original project being imported will build on its own. And because it's vendored, it gives new consumers a clean way to copy the dependency in to their vendor folder in their repo and make use of it.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 30, 2019

Or you now need to have some sort of process to continually check in your module cache to source control.

Yes, but you already need some sort of process to continually check in your dependencies to the vendor directory.

How do you then use that on different developer workstations and build systems?

You provide a GOPROXY that contains all of the modules you need (or, after #26334, one that provides only the modules you're worried about).

It makes me even more convinced that we should keep the more simple vendor concept as the default.

To be clear, all of the various vendoring approaches are power-user stuff: for the vast majority of users, version tags and go.sum already provide complete reproducibility, and module proxies can provide availability.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 30, 2019

because it's vendored, it gives new consumers a clean way to copy the dependency in to their vendor folder in their repo and make use of it.

That is exactly what we don't want to happen. You should only need to store one copy of a given version of a module — not one copy per module you depend one, or one copy per module containing a package main. If a dependency needs to be durable, then it needs to be durable for everyone — not just for folks who happen to have stored it to their vendor directory ahead of time.

@theckman

This comment has been minimized.

Copy link
Contributor

theckman commented Jan 30, 2019

Yes, but you already need some sort of process to continually check in your dependencies to the vendor directory.

Ideally that would be go mod, and it's only happening within the scope of the repository on a developer's workstation. It's not needing to manage and merge the cache across different workstations, build/CI systems, etc.

You provide a GOPROXY that contains all of the modules you need (or, after #26334, one that provides only the modules you're worried about).

What if the GOPROXY is down or has suffered from complete data loss? It's more complexity / risk introduced with the build process, where vendor absolves the risk and simplifies it. No need for external network calls.

To be clear, all of the various vendoring approaches are power-user stuff: for the vast majority of users, version tags and go.sum already provide complete reproducibility, and module proxies can provide availability.

I am not aligned with this approaching power-user stuff, because there are non-power users who value what vendoring brings and appreciates the class of risk it mitigates. I'm also not agreeable to separating "reproducibility" in to "reproducible" and "available" If it's not available it's not reproducible.

@theckman

This comment has been minimized.

Copy link
Contributor

theckman commented Jan 30, 2019

That is exactly what we don't want to happen. You should only need to store one copy of a given version of a module — not one copy per module you depend one, or one copy per module containing a package main.

@bcmills I agree that you should only store one version of the module. I'm instead talking about the case where the module is removed from the Internet, and it can no longer be retrieved from the original location. The only place to get it from is via existing copies (i.e., forks or vendored versions). Ideally you would create a new repo out of the code "saved" from vendor or another fork, and then use a replace statement to use it, versus just injecting it straight in to vendor.

If a dependency needs to be durable, then it needs to be durable for everyone — not just for folks who happen to have stored it to their vendor directory ahead of time.

Unfortunately, that's just not possible unless the go mod tooling decides to go so far as to depend on and use code from its dependency's vendor directories. My initial gut feeling is 👎 for a few reasons, the largest being the perceived complexity it would introduce.

I feel like vendor provides the best effort at giving you the ability to have completely reproducible builds, and a way to resolve missing dependencies if you absolutely need to make use for something. We've had this happen to us a few times now, and those who used vendoring were saved. If you didn't use vendoring, and didn't have it in your module cache, you're going to need to find a friend with the right module cache.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 31, 2019

you're going to need to find a friend with the right module cache.

That is one of several possible roles for the “module mirrors” described in https://blog.golang.org/modules2019, and mirrors seem much more ergonomic than expecting individuals to retrieve and install modules in vendor.

@iand

This comment has been minimized.

Copy link
Contributor

iand commented Jan 31, 2019

My experience report: I'm using a cloud hosted CI system (CircleCI, but could be Travis or Drone) and I want it to build my monorepo using exactly the same code I have on my workstation without it breaking when an upstream dependency is unavailable or has rewritten git history.

I also expect to be able git bisect my monorepo and reproduce exactly the build I had before at that point in the commit history.

Currently a vendor directory checked into my source control system satisfies this requirement exactly. Originally I managed the vendor directory with govendor and then moved to dep. I expect modules + vendoring to support this model.

@aarzilli

This comment has been minimized.

Copy link
Contributor

aarzilli commented Jan 31, 2019

mirrors seem much more ergonomic than expecting individuals to retrieve and install modules in vendor.

This I strongly disagree with. What's more ergonomic, this:

$ go mod vendor

or this:

$ docker pull gomods/athens:v0.2.0

$ docker run -p 3000:3000 gomods/athens:v0.2.0

# from another terminal window

$ git clone https://github.com/athens-artifacts/walkthrough.git

$ cd walkthrough

$ GO111MODULE=on GOPROXY=http://localhost:3000 go build

Not shown: the instructions to install and configure docker.

With the former the only thing I need to do to insure reproducible builds is commit the vendor directory, with the latter I also have to:

  • keep the athens proxy up to date

  • figure out where athens stores the actual cache and set up a backup strategy for it

  • hope athens doesn't change its storage format in the next 20 years

Using a mirror provided by someone else is, of course, remarkably easier but it doesn't give me the same level of build reproducibility because:

  1. it doesn't work if I'm offline

  2. it doesn't work if I'm in a country that happens to be cut off from the mirror for geopolitical reasons

  3. it probably won't work in 10 years

I don't know for sure that Athens and the upcoming Google modules mirror won't last 10 years, but let's be realistic, your companies don't have a good track record for keeping services like this online on that timescale. Google Code, for example, lasted 8 years. This morning your company sent me an email reminding me that Google+ is not long for this world; and that, unlike the modules mirror, was a project that your company was fully behind just 5 years ago.

My personal opinion is that people entrusting mirrors with their project reproducibility are making a mistake. I appreciate that you, and others, don't see it this way, let's agree that only time will tell who's right. I hope I can convince you, with this, that there are reasons to prefer vendor directories over mirrors.

For the reasons above, I believe that there should be a simple way to insure that the vendor directory is sufficient to build the project. I don't care whether this solution or #29058 gets implemented, both guarantee fully reproducible builds (#29058 does it automatically, this one does it by making the build fail unless you remember to run go mod vendor) which is good enough for me.

@balasanjay

This comment has been minimized.

Copy link
Contributor

balasanjay commented Jan 31, 2019

I think points 1 and 2 are resolved by simply replacing git checkout $URL with git checkout $URL && go mod download. The local cache will be deduplicated across multiple projects (no more downloading Gorilla over and over again).

Point 3 is covered by making sure that there exist mirrors being run by multiple independent entities; for instance, I think JFrog recently announced one. And checking that these multiple entities are using some form of mutual replication. (This actually is probably improving the availability of your dependencies over that of your source code repository).

@aarzilli

This comment has been minimized.

Copy link
Contributor

aarzilli commented Jan 31, 2019

I think points 1 and 2 are resolved by simply replacing git checkout $URL with git checkout $URL && go mod download

Those three points were specifically a criticism of using mirrors for reproducible builds, using go mod download has different issues. Specifically, with go mod download I have to archive and backup the module cache alongside each project and since it's shared between al projects and there isn't a good way to prune it, if I actually start to use it for reproducibility it will have to grow indefinitely.

@nomad-software

This comment has been minimized.

Copy link

nomad-software commented Jan 31, 2019

@bcmills

That is one of several possible roles for the “module mirrors” described in https://blog.golang.org/modules2019, and mirrors seem much more ergonomic than expecting individuals to retrieve and install modules in vendor.

What if you are not connected to the internet, the mirrors are down or the company hosting them just randomly decides to not make them available anymore? A vendor folder solves all these issues.

@komuw

This comment has been minimized.

Copy link
Contributor

komuw commented Jan 31, 2019

That is one of several possible roles for the “module mirrors” described in https://blog.golang.org/modules2019, and mirrors seem much more ergonomic than expecting individuals to retrieve and install modules in vendor

We should not forget about all the people that use Go from countries where access to certain websites are restricted.
Such users are going to have an easier time building code that is in their vendor directories than code distributed by mirrors that their governments deem 'inappropriate'

#28652

@typeless

This comment has been minimized.

Copy link

typeless commented Feb 1, 2019

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

As I understand it, Go modules expects there is a single source of truth that is go.mod/go.sum.
If there is a second source of truth from vendor, as the above comment implies, that would be a bit of tricky as the both "truths" have to be synchronized by some rules/mechanism. I think that would probably not be a clean model as it currently is (all dependent sources downloaded locally can be seen as "cached data" and their actual location can be abstracted away).

So, part of the problem is whether Go modules should expect vendor to be modified by tools/users other than Go modules itself. If that's not allowed, the 'implicit replace' problem could be avoided. If that's allowed, things could be more complicated, and I think users would expect their debugging code added to vendor/ should just work.

Considering all that, I am inclined to limit the things users could do to the existing entries in vendor, which is similar to Git does where all existing objects are immutable but you can freely add new ones.
If they need to debugging by modifying the vendored dependencies, they can either add a temporary package under vendor or somewhere else with the corresponding replace statement, and the consistency between go.sum/go.mod and the actual code wouldn't be affected.

Finally, I am a user of vendor and have a pending PR for migrating Gitea to modules. The PR uses -mod=vendor to make it work currently. But eliminating vendor (or an alternative like a "proxy") would be a non-starter for the project.

@balasanjay

This comment has been minimized.

Copy link
Contributor

balasanjay commented Feb 1, 2019

@aarzilli I was suggesting using mirrors and using go mod download, if you needed to solve those 3 given points.

And I don't see why you think growing indefinitely is a unique problem to go mod download, given that vendoring also grows your repository without bound (and likely your disk space too for all commonly used VCSs). Especially given vendor dirs are not shared across projects.

@provPaulBrousseau

This comment has been minimized.

Copy link

provPaulBrousseau commented Feb 4, 2019

I may not have a complete breadth of knowledge (I've been wrestling with the tools a little), but this is my use & expectation of vendoring.

We have a small monolith repo of related Go projects, with a single vendor directory. We are using go.mod to manage versions. We are vendoring all our 3rd party packages and build tools (go-bindata and gomock at the moment). The goal is to have 100% consistent builds both locally and in our CI environment, as well as 100% reliability (durability against dependencies going away, working offline). It is also my hope that grabbing a shallow git clone will speed up the build process because all required resources will already be present.

The project that I am working on is to build tools to help our internal developers manage K8S / helm deployments. We are a team of 5 at the moment, supporting (guessing) 50 developers.

@ssoroka

This comment has been minimized.

Copy link

ssoroka commented Mar 19, 2019

I'm shocked that -mod=vendor isn't the default behavior. Why would I create a vendor folder and then not want it used? Let's follow the principle of least surprise here, and if the vendor folder exists, assume it's intended to be used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.