Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: automatic vendoring in module mode #30240

Open
bcmills opened this Issue Feb 14, 2019 · 53 comments

Comments

Projects
None yet
@bcmills
Copy link
Member

bcmills commented Feb 14, 2019

This proposal overlaps with (and hopefully unifies) several existing issues, linked in the text below.

I'd like to implement it soon, in the 1.13 cycle, so if you have feedback please do respond quickly. 馃檪

Problem summary

Users want a durable, local view of their source code that works with existing diff tools and does not require per-user configuration in cloned repositories.

  • Relying on module proxies does not necessarily satisfy delivery contracts.
  • Saved module caches do not interoperate well with version-control and code-review tools.
  • -mod=vendor requires configuration per user (GOFLAGS) or per invocation, and makes it too easy to ship code that produces a different build in vendored mode than in the normal module mode.

Proposal

Under this proposal, the source code for the packages listed in vendor/modules.txt 鈥 and the go.mod files for the modules listed in vendor/modules.txt, if any 鈥 will be drawn from the vendor directory automatically (#27227).

If a replace directive in the main module specifies a module path, the module source code will be vendored under the path that provides the replacement, not the path being replaced. That preserves the 1:1 correspondence between import paths and filesystem directories, while allowing replacement targets to alias other modules (#26904). If a replace directive specifies a file path, then either that path must be outside the vendor directory or the vendor/modules.txt file must not exist (#29169).

Package patterns such as all and example.com/... will match only the packages that are present in the vendor directory, not unvendored packages from the same module. During the build, if additional packages from the vendored modules are needed in order to satisfy an import, the source for those packages will be fetched (from the module cache, if available) and added to the vendor directory. (Packages from outside the already-vendored modules will not be vendored automatically.)

Any time the go.mod file is written, if a module path found in vendor/modules.txt has a different version than that found in the build list, the already-vendored packages and go.mod file from the previous version will be deleted, and updated versions of those packages will be written in their place (#29058). Transitive imports of those packages will be resolved, and may populate additional packages in other already-vendored modules.

If go get removes a module from the build list entirely, its package source and go.mod file will be removed, but an entry for the module (with version none) will remain in vendor/modules.txt. That way, if a future operation (such as a go get or go build) adds the module to the build list again, it will remain vendored as before.

When go mod tidy is run, it will add or remove packages from the vendor directory so that it continues to contain only the subset of packages found in the transitive import graph. It will also remove go.mod files and entries in vendor/modules.txt for modules that are no longer present in the build list.

To encourage the minimal use of vendor directories, the go mod vendor subcommand will accept an optional list of packages or modules. go mod vendor <module> will update the vendor directory to contain the go.mod file for <module> and source code for its packages that appear in the transitive import graph of the main module. (Note that, since the criterion for inclusion of a package is its existence in the import graph, vendoring in an additional module should not affect the contents of any previously-vendored modules.)

go mod vendor <pattern> for an arbitrary module pattern will add # <pattern> to vendor/modules.txt, and vendor in the go.mod files (and any packages found in the import graph) for modules matching <pattern>, adding individual comments to vendor/modules.txt for those modules.

Note in particular that go mod vendor all will copy in go.mod files for all of the module dependencies in the module graph (and add entries in vendor/modules.txt for those modules). That ensures that after go mod vendor all, go list can produce accurate results without making any further network requests (see also #19234 and #29772).

The go mod vendor subcommand will accept a new flag, -d. go mod vendor -d <pattern> will remove all previously-vendored modules matching <pattern> from the vendor directory (and from vendor/modules.txt), as well as any previously-stored patterns matching those modules (including <pattern> itself, if present).

go mod vendor, without further arguments, is equivalent to go mod vendor all. go mod vendor -d is equivalent to go mod vendor -d all. If go mod vendor -d causes vendor/modules.txt to become empty, it will also remove the entire vendor directory.


Edits

@bcmills bcmills added this to the Go1.13 milestone Feb 14, 2019

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 14, 2019

@FiloSottile

This comment has been minimized.

Copy link
Member

FiloSottile commented Feb 14, 2019

To encourage the minimal use of vendor directories, ...

Why are partial vendor folders something we want to encourage? Most use cases listed here and in the linked issues would require all dependencies vendored at all times.

Also, can you clarify if go get or go mod tidy would ever add new modules to the vendor folder, or if running go mod vendor would still be required after every new dependency is added in order to avoid a partial vendor folder?

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 14, 2019

Why are partial vendor folders something we want to encourage? Most use cases listed here and in the linked issues would require all dependencies vendored at all times.

Some dependencies are more robust than others. For example, you might trust github.com to be generally available, but want to vendor in dependencies that happen to be hosted using bzr or svn so that you don't have to install those tools on every machine that will build your module.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 14, 2019

Also, can you clarify if go get or go mod tidy would ever add new modules to the vendor folder, or if running go mod vendor would still be required after every new dependency is added in order to avoid a partial vendor folder?

go get and go mod tidy would not add dependencies to vendor automatically.

We could perhaps make go mod vendor (without arguments) set some flag in modules.txt to indicate that all additional modules should be vendored.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 14, 2019

More generally, though, the main goal of automatic vendor updates is to prevent version skew. Copying in newly-added modules does not further that goal, since there are no out-of-date contents in the first place.

@thepudds

This comment has been minimized.

Copy link

thepudds commented Feb 14, 2019

I would suspect the most common use case might be vendoring 100% of dependencies?

If so, and if vendoring in 1.13 is going to be able to track updates via go get and go mod tidy in some cases, it would seem that once you have signaled you want automated tracking it likely should be the default behavior at that point to be 100% complete in any automated tracking, rather than defaulting to partial tracking? (For example, track all updates after a go mod vendor with no args, as you suggested two comments back)?

@FiloSottile

This comment has been minimized.

Copy link
Member

FiloSottile commented Feb 14, 2019

It definitely makes sense to support partial ones. I just suspect (and might be wrong!) that 90% of users opting into vendoring really mean to vendor everything, and a reasonable chunk of that 90% would be surprised by it behaving otherwise.

@ianthehat

This comment has been minimized.

Copy link

ianthehat commented Feb 14, 2019

In the presence of a reliable proxy, I can't think of any reasonable cases for a partial vendor directory, and lots of possible confusion.
I would personally argue we go in the other direction, as in if you try to build in vendor mode it is not allowed to see anything outside the current module (except the stdlib)

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 14, 2019

@ianthehat, one use-cases for vendoring, given proxies, is to vendor in private code for which the proxies do not have access.

For example, a contract-based startup might want to vendor in their proprietary utility modules before delivering the code to their customers.

@thepudds

This comment has been minimized.

Copy link

thepudds commented Feb 15, 2019

@bcmills Could you comment on the interplay with -mod=readonly, and/or options to disable automatic downloads for people who would prefer to fail if vendor is missing something?

@ianthehat

This comment has been minimized.

Copy link

ianthehat commented Feb 15, 2019

@bcmills you can achieve the same effect by copying to an internal package and rewriting the import paths, which would be more honest, and also allow for local modifications (something else that those kinds of contractors often need to do as well). If you don't want to rewrite the import paths, you could check it in as a sub-module and use a replace directive (you probably have full control of the main go.mods for that kind of work)
Or you could add a directory with the zip and mod files and use it as a file proxy (which is something it might be worth looking into as a better version of vendoring)
I don't think making the normal use much worse for such extreme edge cases would be the wrong choice.

@lopezator

This comment has been minimized.

Copy link

lopezator commented Feb 15, 2019

I like removing complexity, flags, and per user (or per project) configuration when using vendored mode. I think that automatic detection of vendor folder (and assume you are in vendored mode) when a vendor folder is present it's a great idea.

I sometimes mix vendored and non-vendored projects, and switching between would be great to be as transparent as possible.

I agree though with the opinion of some of the folks above, IMHO supporting partial vendoring would be confusing and it will add complexity.

For example, in our usecase, we are using non-vendored mode for our main projects, adding a GOPROXY for public libraries, but don't want to cache our private libraries there (for security, and because cache server and source server are on the same local network, it just doesn't add any benefit for us). #26334 would be enough for this.

Vendored mode, in the other hand, it's great to distribute self-contained/small apps/tools.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

@ianthehat

I would personally argue we go in the other direction, as in if you try to build in vendor mode it is not allowed to see anything outside the current module (except the stdlib)

Part of the point of this proposal is to avoid the need for a distinct 鈥渧endor mode鈥. Modules are integrated into the normal go workflow, and if we're serious about supporting vendoring, then I would argue that vendoring should be integrated too.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

Or you could add a directory with the zip and mod files and use it as a file proxy (which is something it might be worth looking into as a better version of vendoring)

We've considered that, but it really doesn't work well with version control systems: the diffs are incomprehensible and the blobs can end up consuming a lot more space than they ought to (depending on the encoding).

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

Re partial vendoring: given module proxies, the major use-case for vendoring is for modules that are not available via the public proxies. (Recall that the word 鈥渧endor鈥 literally means 鈥渙ne who sells鈥.)

Module mode substantially reduces the need to duplicate code: you no longer have to copy all of your dependencies into your own repository, and you especially don't need to do that for stable, publicly-available, open-source dependencies. It is important to me that we make it easy to duplicate the minimum amount of code necessary for each use-case: minimal duplication shouldn't be an 鈥渆xtreme edge [case]鈥, it should be the default mode of operation.

It's not realistic to expect folks to manually apply replace directives for partial vendoring, or to rewrite import paths. It's certainly possible, but it's extremely tedious (see #30241 and #27542). It isn't, and shouldn't be, a default mode of operation. If that were the only alternative to vendoring the full tree, folks wouldn't do it: instead, they'll fall back to duplicating all of the dependencies all over again.

The point of vendoring in module mode is not to provide an alternative to using modules. It is to provide a complementary feature set for the cases that modules cannot address well: namely, the distribution of proprietary code.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

That said, let's think about that sticky-pattern problem. I don't buy the 鈥渇ull vendoring as a default鈥 argument, but there is a more general case that really ought to work.

Suppose that I run go mod vendor golang.org/x/.... I should reasonably expect any further dependencies matching golang.org/x/... to be vendored.

If we support that, then we can view go mod vendor without arguments as equivalent to go mod vendor all, and that will provide sticky full-vendoring.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

So how about this alternative. For a given module pattern,

  • go mod vendor <pattern>
    • adds # <pattern> to vendor/modules.txt, and
    • vendors in the go.mod files (and any packages found in the import graph) for modules matching <pattern>, adding individual comments to vendor/modules.txt for those modules.
  • go mod vendor -d <pattern> removes from vendor/modules.txt:
    • <pattern> itself, if present;
    • all modules matching <pattern>;
    • and finally, all further patterns that match the removed modules.

And then go mod vendor is defined to be equivalent to go mod vendor all.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

@thepudds

Could you comment on the interplay with -mod=readonly, and/or options to disable automatic downloads for people who would prefer to fail if vendor is missing something?

Under this proposal, -mod=readonly would continue to disable updates to the go.mod file, but any imports already listed in vendor/modules.txt that are found during a go build would be copied into the vendor directory.

-mod=vendor would continue to exist, and would mean 鈥渄o not resolve imports that are not found in either GOROOT or vendor鈥. However, since we would now vendor in go.mod files as well, go -mod=vendor would produce more accurate results from subcommands like list, mod why, and mod graph that examine the structure of the module graph.

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Feb 15, 2019

So how about this alternative. For a given module pattern,

  • go mod vendor <pattern>

    • adds # <pattern> to vendor/modules.txt, and
    • vendors in the go.mod files (and any packages found in the import graph) for modules matching <pattern>, adding individual comments to vendor/modules.txt for those modules.
  • go mod vendor -d <pattern> removes from vendor/modules.txt:

    • <pattern> itself, if present;
    • all modules matching <pattern>;
    • and finally, all further patterns that match the removed modules.
      And then go mod vendor is defined to be equivalent to go mod vendor all.

I think this works very well for me, thanks. I couldn't reason through all the cases you listed in your original post (I'll try to go through them over the weekend), but surely this command line API looks good and the sticky mode is really good.

Is there really a need to introduce a third metadata file (vendor/modules.txt), after go.mod and go.sum? Did you think of adding a vendor command to go.mod?

@thepudds

This comment has been minimized.

Copy link

thepudds commented Feb 15, 2019

@bcmills In addition to the proposed new behavior described above, is the thinking that this would also land in 1.13:

  • #27348 "cmd/go: allow verifying vendored code"

If so, under the latest proposal, is this an example of what a module author could do if they want to fail if vendor is incomplete:

  1. in the author's own builds or in their CI, they could run with -mod=vendor to fail if vendor is incomplete
  2. for consumers, the author does not have control over what consumers do (and relying on a README stating "please set -mod=vendor" is not a desired solution). However, if the author runs go mod vendor (no args), that provides a complete vendor directory on an on-going basis based on the proposed automatic tracking behavior, and in addition the author could run go mod verify -vendor (or go mod vendor -verify or whatever incantation) to verify that vendor is both correct and complete? And if go mod verify -vendor is successful (say, prior to releasing a new version of a module), the author would have confidence that a consumer would never automatically download new code to populate vendor (even if the consumer is not running with -mod=vendor or -mod=readonly)?
@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

Is there really a need to introduce a third metadata file (vendor/modules.txt), after go.mod and go.sum? Did you think of adding a vendor command to go.mod?

I hadn't really considered it: I think @rsc added vendor/modules.txt in 1.11, and given that it's already there I figured we could keep using it.

I suppose that we could record the patterns in go.mod instead, but I have a mild aesthetic preference for keeping them in modules.txt. I'm certainly open to arguments to the contrary, though. 馃檪

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

Updated the proposal to incorporate sticky patterns (#30240 (comment)).

@ianthehat

This comment has been minimized.

Copy link

ianthehat commented Feb 15, 2019

We've considered that, but it really doesn't work well with version control systems: the diffs are incomprehensible and the blobs can end up consuming a lot more space than they ought to (depending on the encoding).

If your use case is because the code cannot live in a public proxy, why do you care about the diff, you would not see the diff if it was in the public proxy. It's also trivial to fix, use a non compressed text archive. This also fixes the space issue.

@ianthehat

This comment has been minimized.

Copy link

ianthehat commented Feb 15, 2019

Part of the point of this proposal is to avoid the need for a distinct 鈥渧endor mode鈥. Modules are integrated into the normal go workflow, and if we're serious about supporting vendoring, then I would argue that vendoring should be integrated too.

I think we ought to start by enumerating the actual problems we are hoping to solve with vendoring, and checking it is the right solution to those problems. Vendoring comes with a lot of serious problems, it needs to be worth the cost.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

If your use case is because the code cannot live in a public proxy, why do you care about the diff, you would not see the diff if it was in the public proxy.

For a start, if you're vendoring the code because it is proprietary, you want to be sure that you are shipping only what was actually promised to the customer.

(In contrast, if the module is already publicly available, you probably don't care which parts you're re-publishing in your vendor directory.)

It's also trivial to fix, use a non compressed text archive.

That is essentially what the vendor directory is: it just happens to be text archive format that can also be consumed by pre-module versions of the go command.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

To make sure I understand correctly, that would include any future packages that you obtain via go get ...?

Yes, or any future packages that you obtain by adding an import of a package and letting go build resolve that import.

Basically, under this proposal (with 鈥渟ticky vendoring鈥), go mod vendor <pattern> tells the go command to always keep the vendor directory up-to-date for everything matching <pattern>.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

@ianthehat

you are talking about adding features to the go command, to use them you inherently must have a go command that supports modules, and that's our story for reproducible builds.

I'm talking about adding features to manage the vendor directory. Previous versions of the go command back to 1.6 can consume the vendor directory, and (unless I have made a serious mistake, which is possible) the directory tree created under this proposal remains compatible with them.

@theckman

This comment has been minimized.

Copy link
Contributor

theckman commented Feb 15, 2019

That means that 鈥渨ithout the need for a highly available proxy鈥 is equivalent to 鈥渨ithout the need for a reliable filesystem鈥, and if your CI system does not have a reliable filesystem you're probably not going to have a good time with go build anyway.

@bcmills if I build inside of containers, I will intentionally not have a reliable filesystem within CI. It'll be completely blown away after the resulting binary is yanked out of the container.

(But that's mostly irrelevant, because we plan to have highly available public proxies anyway: see https://blog.golang.org/modules2019.)

Google, much like NPM, has not been 100% immune to service-interrupting issues. It's the nature of the business, and so I don't feel your comment is a valid counter-point. People will deploy bad code or configuration, systems will fail, etc.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

  • Unpacking source code for temporary auditing (such as using grep -r) or ephemeral debugging (such as inserting println statements).

This is exactly what the replace directive is supposed to be for, if it does not work well, lets fix it, not suggest vendoring as an alternative.

If you like, you can think of vendor/modules.txt as defining a set of replace directives, along with a set of rules to update those replace directives based on future changes to the build list.

That's actually where I started this design: with the question, 鈥渨hat would it look like if we unified vendoring and replacements?鈥

The interesting part is the automatic-update step, since we want the auditable form of the code to stay in sync with the actual modules in use. vendor is what you get if you produce the source code from the modules; replace is what you get if you produce the modules from the source code.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

@theckman

Google, much like NPM, has not been 100% immune to service-interrupting issues. It's the nature of the business, and so I don't feel your comment is a valid counter-point. People will deploy bad code or configuration, systems will fail, etc.

I would be very surprised if Google were the only organization maintaining a public module proxy. If Google's proxy goes down, you can set GOPROXY to point to someplace else and keep on working.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

@ianthehat

the vendor directory is real code that you can modify, have no idea if it has been modified

That's #27348, and I expect we'll address it by 1.14 (if not yet in 1.13).

@theckman

This comment has been minimized.

Copy link
Contributor

theckman commented Feb 15, 2019

@bcmills Now that introduces further complexity and additional trust issues around that proxy and whether they are safe for use, further convincing me that my call-out for use case is valid. If the main official proxy goes down, I'm confident the overwhelming majority of users are not going to spend the development cycles to identify a trusted alternative proxy, and to then update all of their CI configurations to use it.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

@theckman, the issues of proxy trust are certainly valid, and they will be addressed in-depth in other proposals for Go 1.13. Stay tuned!

However, those details are a bit off-topic for this proposal.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 15, 2019

At any rate: while I don't agree that reproducible builds are a compelling use-case for vendoring, the current proposal supports that use-case anyway.

We can (and, I hope, will) come to agreement on a design even if our underlying reasons for it differ.

@rasky

This comment has been minimized.

Copy link
Member

rasky commented Feb 15, 2019

  • Shipping proprietary dependencies to customers under contract.

This is the interesting one, and I still think vendoring is totally the wrong choice for this.
We redestribute modules using a proxy, this is just a custom proxy, so lets do it that way. Vendoring is a horribly confusing to the user way to distribute a module.

I think we have different users. With vendoring support, my customers don't even need to be aware what vendoring is. I provide a git repository, they clone it and run "go build"; everything works, and everything is self-contained (shipping all the dependencies in source code format is part of many of my contracts, and vendoring allows me to honor this clause very easily and effortlessly).

You seem to suggest that your customers/users are happy with setting up "custom proxies" (whatever that means) in addition to cloning the project for the purpose of building it, possibly using additional documentation that you have provided them with. I'm happy it works for you, and I can assure you that you will not find me passionately arguing against custom proxies in GitHub issues just because I don't use them.

If you don't have a use case for vendoring, please rest assured that there are many people who do, and there have been multiple discussions going on on GitHub issues and directly with Google to try to have proper vendoring support with modules as well. If you feel so strongly, please open a proposal to remove vendoring from the Go toolchain, but I think it's better not to hijack this issue by arguing on the actual merits of vendoring.

@thepudds

This comment has been minimized.

Copy link

thepudds commented Feb 15, 2019

@theckman @bcmills As far as I can see, I think you might be agreeing with each other that you both seem to view that a filesystem-based module cache that is checked into VCS is not a valid substitute for a traditional vendor directory.

To evaluate the proposal here, it is valuable to spell out use cases serviced by a traditional vendor directory and evaluate those against future proposed behavior, and you can describe and compare behaviors of traditional vendoring, etc.

But at least for me, if you wanted to try to put it in a single sentence, the most important part of evaluating whether or not a future solution is a valid substitute for a traditional vendor directory might be:

Allow tracking third-party code in VCS in a similar manner to how you track your own code.

(Different people might want that for different reasons. For example, some people might think some set of the following: VCS is the ground truth for your source code and should be the ground truth for other code you use or ship even if someone else wrote it; greater trust in your VCS than any other external system; VCS is more likely to survive over longer timescales or migrate forward in the face of change; the desire to use standard VCS tools (diff, tagging, blame, bisect, etc.) on third party code in a similar manner to how you use them with your own code; building when disconnected from the public Internet; supporting reproducible builds; internal policy; compliance; auditing; etc.).

My personal view is a file-based proxy or module cache checked into VCS does not meet that threshold. One example is a file-based proxy I think has N copies of different versions of a dependency under file directories like v1.2.3, v1.2.4, etc., which is not how you track your own code in VCS. That then has different implications for how you interact with that dependency code.

But I also suspect different people would boil it down to a different single sentence, and perhaps the topic is too nuanced to try to capture in a single sentence.

@ianthehat

This comment has been minimized.

Copy link

ianthehat commented Feb 15, 2019

I think we have different users. With vendoring support, my customers don't even need to be aware what vendoring is. I provide a git repository, they clone it and run "go build"; everything works, and everything is self-contained (shipping all the dependencies in source code format is part of many of my contracts, and vendoring allows me to honor this clause very easily and effortlessly).

You seem to suggest that your customers/users are happy with setting up "custom proxies" (whatever that means) in addition to cloning the project for the purpose of building it, possibly using additional documentation that you have provided them with. I'm happy it works for you, and I can assure you that you will not find me passionately arguing against custom proxies in GitHub issues just because I don't use them.

Sorrry, I was not clear.
I am suggesting that checking in a proxy directory that the go command knows how to automatically use would be a superior solution to checking in a vendor directory if this was the only concern. It meets all the requirements for a stand alone build in a way that unifies with modules cleanly, and has none of the drawbacks of a vendor directory.

@kardianos

This comment has been minimized.

Copy link
Contributor

kardianos commented Feb 15, 2019

@bcmills I agree with your problem summary and proposed solution. This is a great proposal. I went back and forth on the module pattern. I think it is more then athletics to put the patterns in modules.txt, as it pertains only to the act of vendoring, and not to module or go versions. So I agree with your proposal on that as well fully.

Thank you. This will make modules even easier to use for myself and my team.

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Feb 20, 2019

Change https://golang.org/cl/162989 mentions this issue: go/analysis: allow overriding V flag without code patches

gopherbot pushed a commit to golang/tools that referenced this issue Feb 20, 2019

go/analysis: allow overriding V flag without code patches
In CL 149609, a file was added to
src/cmd/vendor/golang.org/x/tools/go/analysis/internal/analysisflags/patch.go
to override the behavior of the V flag for cmd/vet.

That modification causes the behavior of cmd/vet to change when a
pristine copy of x/tools is vendored in, and module-mode vendoring
will only support pristine copies (see golang/go#30240).

Instead, allow cmd/vet to override the V flag by defining its own V
flag before it invokes unitchecker.Main.

Tested manually (by patching into cmd/vendor).

Updates golang/go#30240
Updates golang/go#30241
Updates golang/go#26924
Updates golang/go#30228

Change-Id: I10e4523e1f4ede94fbfc745012dadeefef48e927
Reviewed-on: https://go-review.googlesource.com/c/162989
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Feb 23, 2019

This proposal has an interesting interaction with #27852. If we start storing go.mod files in vendor directories, then nothing in the vendor directory will be included in the module cache.

That's probably fine, because building within the module cache doesn't work in general anyway (modules may have replace directives pointing to files outside of the module), but 鈥 that being the case 鈥 we could trim out a lot of code bloat in the module cache by explicitly excluding the vendor directories associated with cached modules.

@driusan

This comment has been minimized.

Copy link

driusan commented Mar 4, 2019

This all makes sense for "package main" type modules, but what happens if you run go mod vendor under a library package with this proposal?

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Mar 4, 2019

@driusan, why would it need to be any different for modules containing libraries?

(Note that in module mode, we only ever use the vendor directory of the main module. Packages vendored by dependencies are not searched and not relevant to the build: that way, the main module has complete control over which dependency versions are in use, and there is only one non-test copy of the source code in use for each package import path.)

@driusan

This comment has been minimized.

Copy link

driusan commented Mar 4, 2019

@bcmills Your note is why I think it should be different (and should probably result in an error).. if it behaves the same way regardless of whether it's a main module or a library module, it would result in spurious vendor directories that are never used (under any circumstances) when run under a library package, which is probably not what the user intended.

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Mar 4, 2019

The vendor directories would still be used when working within that library module itself, and when compiling the program in GOPATH mode (for example, using an older version of the Go toolchain).

The use-cases for vendoring in library modules are a bit less compelling than for modules that include main packages, but as far as I can tell that difference mostly only affects the decision about whether to use a vendor directory at all 鈥 not what it should do if present.

(Note that the inclusion of go.mod files under this proposal will tend to exclude the vast majority of vendor directory contents anyway.)

@gopherbot

This comment has been minimized.

Copy link

gopherbot commented Mar 5, 2019

Change https://golang.org/cl/165378 mentions this issue: all: add -mod=vendor to GOFLAGS in tests that execute 'go' commands within std or cmd

gopherbot pushed a commit that referenced this issue Mar 5, 2019

all: add -mod=vendor to GOFLAGS in tests that execute 'go' commands w鈥
鈥thin std or cmd

Updates #30228
Updates #30240
Updates #30241

Change-Id: Idc311ba77e99909318b5b86f8ef82d4878f73e47
Reviewed-on: https://go-review.googlesource.com/c/go/+/165378
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@selslack

This comment has been minimized.

Copy link

selslack commented Mar 7, 2019

@bcmills
My experience report:
I vendor all the dependencies, I do not modify the vendor folder (I do not edit or patch sources there). My build is offline and fails if I have a missing dependency. No magic, I'm happy.

My expectations from new tools provided by Go itself:

  1. Keep the old behavior of the vendor folder (which we had for years, literally);
  2. Work outside of GOPATH (which simplifies development).

Please provide a simple and reliable solution which satisfies the community's expectations.

I've read both big threads and I see that there are basically two use cases:

  1. Vendor all the dependencies.
  2. Vendor none.

And one potential use case:

  1. Vendor some.

I haven't seen experience reports with this one (maybe I missed one, these threads are super big). But this potential case smells like nobody likes complexity like setting up a proxy with access to private repositories.

Thanks!

@bcmills

This comment has been minimized.

Copy link
Member Author

bcmills commented Mar 7, 2019

@selslack, thanks for the report, but it would help even more if you could share the why of your current workflow instead of the what. (In particular: why, if at all, would you prefer to vendor packages rather than modules? Why, if at all, would you prefer to vendor source files rather than zipfiles?)

We know how things work today, but we also know that for at least some use-cases there will be better alternatives in module mode. I want to optimize the vendor functionality for the subset of use-cases that don't have a better alternative.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can鈥檛 perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.