Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: exclude vendor dir from matching `...` #19090

Closed
natefinch opened this Issue Feb 14, 2017 · 80 comments

Comments

Projects
None yet
@natefinch
Copy link
Contributor

natefinch commented Feb 14, 2017

There are some closed issues about this, but I'd like to bring it up again.

Any time you're building an application and you make an update to any package in the repo (of which there are often many), you run go test ./... to make sure you haven't broken your program. This worked fine before vendoring became a thing, but with vendoring, it runs all the tests of vendored code as well. While you may want to do that once in a while, such as when you update the code in the vendor directory, it's generally not something you want to do every time your code changes (because it shouldn't affect code you depend on at all).

I'd like to suggest that ... ignore the vendor directory by default. I almost never want to have any command touching the vendor directory directly - not go vet, not go fmt, not go build, not go test, not go install, not go fix.

If the community feels that there is utility in being able to match the vendor directory, I suggest adding a flag called -vendor (or whatever) that indicates ... should match the vendor directory. But by default, I think almost everyone would prefer it not match the vendor directory.

As we start down the path of having an official package management solution, I think vendoring will get more and more common.... and this will become more and more of a pain.

(and yes, I know you can do the go test $(go list | grep -v vendor) thing, but that's shell and OS-dependent, and ugly, and just bad UX)

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Feb 14, 2017

I feel similarly about this issue.

vendor could be ignored when walking its parent, so ./... would ignore it but ./vendor/... wouldn't.

@dlsniper

This comment has been minimized.

Copy link
Contributor

dlsniper commented Feb 14, 2017

I would like to add the votes of 21 people that do not want to have their editor run go test against vendor/ at all https://youtrack.jetbrains.com/issue/GO-2366 and 19 others that don't want gofmt to run against vendor/ as well https://youtrack.jetbrains.com/issue/GO-2412. One could argue that the editor should be smarter and send a list of packages to the tool but then why should users be forced to either have to use $(go list | grep -v vendor), which mind you is not possible on Windows, or use an editor to achieve this? WSL does not count as it introduces yet again more work for the user, thus bad UX.

Also, please notice tools like glide, one popular dependency management tool, where there's even a dedicated command to list packages and omit the vendor directory in order to make life easier.

This is clearly an UX problem that the users should not have to face.

@ianlancetaylor ianlancetaylor changed the title Exclude vendor dir from matching `...` cmd/go: exclude vendor dir from matching `...` Feb 14, 2017

@ianlancetaylor ianlancetaylor added this to the Go1.9 milestone Feb 14, 2017

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 14, 2017

Previous discussion:

  • #11659 exclude vendor directories from wildcard matches?
  • #14417 cmd/go: go test ./... fails on go 1.6 because dependencies are tested
  • #17497 cmd/go: go generate ./... will recurse into vendor
  • #17879 Exclude vendor/ from linters

Marking as Go 1.9 for further consideration and decision.

@dlsniper

This comment has been minimized.

Copy link
Contributor

dlsniper commented Feb 14, 2017

I would also like to add that this is what a Windows user currently has to write when using Windows CMD to work around the vendor issue

for /f "" %G in ('go list ./... ^| find /i /v "/vendor/"') do @go test %G

which is a lot worse than go test $(go list | grep -v vendor).

@ernesto-jimenez

This comment has been minimized.

Copy link

ernesto-jimenez commented Feb 14, 2017

@natefinch thanks for raising this again.

Regarding the usual argument that "./... means all sub packages should avoid making vendor a especial case":

  1. vendor is an especial case.
  2. ./... already ignores folders with names starting with . or _, so there's a precedent supporting this proposal.
@PaulReiber

This comment has been minimized.

Copy link

PaulReiber commented Feb 14, 2017

@dlsniper

This comment has been minimized.

Copy link
Contributor

dlsniper commented Feb 14, 2017

@PaulReiber see this comment: #11659 (comment)

"vendor" is not being renamed. That ship has long sailed.

@ianlancetaylor can you please add #11659 to the list of related issues as hopefully your comment will be more visible than mine. Thank you

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Feb 14, 2017

@dlsniper Done, thanks.

@rakyll

This comment has been minimized.

Copy link
Member

rakyll commented Feb 14, 2017

This was overwhelmingly mentioned in the tools survey I conducted last year. Combining all feedback I got (individuals, mailing list questions, StackOverflow, etc), this particular problem was one of the top problems that has been mentioned.

@johnt337

This comment has been minimized.

Copy link

johnt337 commented Feb 15, 2017

A core function like test that would understand vendoring as a dependency and not the target of testing scope would be huge. It's inefficient to write custom regex patterns to include some portions of code and not others. Increasingly so when you want to operate on subsets of the code base. The ./... Operator forces you to glob all go files with the expressed understanding that you will filter out a majority of the results returned wastes cycles during iterative testing. Given this is a solved problem with tools like glide it would seem like a quick win scenario with tangible returns to the go dev community.

@minux

This comment has been minimized.

Copy link
Member

minux commented Feb 15, 2017

@natefinch

This comment has been minimized.

Copy link
Contributor Author

natefinch commented Feb 15, 2017

That doesn't help for all the rest of the tooling. I don't want go vet complaining about problems in vendored code. I definitely do not want go fmt or go fix changing things in vendored code. I don't want go install installing binaries that happen to be in vendored repos.

The root of it is this: without vendoring, ... never matches dependencies that arent't in your repo, because they're not subdirectories. I just want to maintain that functionality while vendoring.

@minux

This comment has been minimized.

Copy link
Member

minux commented Feb 15, 2017

@dlsniper

This comment has been minimized.

Copy link
Contributor

dlsniper commented Feb 15, 2017

I've also just got bitten by an issue that go vet complains about code in the vendor/ directory but until the upstream changes that code I can't do anything there thus I'm forced to use the workaround instead of not having to deal with it at all.

And I don't think that go fmt ./... is overused when you want to format the project and make sure it's all consistent inside a pipeline for example.

@minux

This comment has been minimized.

Copy link
Member

minux commented Feb 15, 2017

@sparrc

This comment has been minimized.

Copy link

sparrc commented Feb 15, 2017

Then don't invoke go fmt or go fix on ./..., which is very problematic even
if there aren't any vendored packages.

why is this "problematic"? We've never had any problems with it, and we run it on every commit merged into InfluxDB, Telegraf, & Kapacitor.

and I personally don't care if my dependencies are go fmt'ed properly, I only want to run it on my codebase.

Changing the behavior of ./... today to ignore vendor packages will only
introduce confusion.

My experience has been most people being confused by it not ignoring vendor/.

@mibk

This comment has been minimized.

Copy link
Contributor

mibk commented Feb 15, 2017

e.g.
go test ./... -./vendor/... # using - is preferred, but -exclude also
works.

While this concept could definitely be applied to more use cases, excluding vendor is the real pain IMO. For other use cases go test $(go list | grep -v ...) always does the job.

Changing the behavior of ./... today to ignore vendor packages will only
introduce confusion.
(And there are reasonable uses of ./... to match vendor packages.)

If we don't want to break backward compatibility and therefore prevent potential confusion, could it be tolerable by both sides to introduce a single-character flag for excluding vendor? It would be only 3 keystrokes more to type without any BC break.

@hodgesds

This comment has been minimized.

Copy link

hodgesds commented Feb 15, 2017

What about a .gorc file that can be configured with defaults? That way it can be configured globally ( ~/.gorc ) if you hate doing grep -v or set within a package when running tool on vendor/ is desired.

@minux

This comment has been minimized.

Copy link
Member

minux commented Feb 15, 2017

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Feb 15, 2017

@minux why would you want to explicitly install vendored packages? If they are vendored, they are dependencies and will already be built if vendor is ignored.

@sparrc

This comment has been minimized.

Copy link

sparrc commented Feb 15, 2017

Please note that we do want to build/install the vendored packages

@minux, that's not true, in telegraf I only want to build the packages in vendor/ that telegraf depends on, not every single package in vendor/.

Building every package in vendor/ means that you would have to trim out packages/tests that your project doesn't use.

This is not how it works when your dependencies are in your GOPATH.

@minux

This comment has been minimized.

Copy link
Member

minux commented Feb 15, 2017

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Feb 15, 2017

If you vendor only necessary packages, then you could use go install ./....

True, but it still doesn't explain why you need the vendored packages to be included. This just explains a way for it to not matter.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Feb 15, 2017

Rather than expressing incredulity that people want to use the glob operator, I think it is important to recognise that globbing is a straight forward solution to very commonplace questions like "How can i run the tests for all the packages in my project?", which, until go 1.6, was satisfied by globbing.

Saying that people shouldn't use the glob operator is not helpful as the replacement was non portable (how do you pipe grep on windows?), or relies on an external tool like go test $(glide nv); which also isn't a portable solution.

As @rakyll showed from her research this issue is a major point of confusion for Go developers and that alone should be cause to reexamine the decision.

@iand

This comment has been minimized.

Copy link
Contributor

iand commented Feb 15, 2017

Another consideration is that the current behaviour requires you to vendor dependencies of your dependency's tests, e.g. packages like testify

@jsternberg

This comment has been minimized.

Copy link

jsternberg commented Feb 15, 2017

Related to the above consideration, go get ./... will not bring in those dependencies automatically so you have to go get ./..., vendor the dependencies, then repeat until no new vendored dependencies show up. While it's easy enough to make a custom tool that does this, it seems like it should be unnecessary for the default go toolchain to cause this problem.

It seems to me like you should be able to call go get ./..., vendor the dependencies identifies and retrieved by the command, and go test ./... would still work. This isn't currently true.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Feb 16, 2017

@jsternberg if possible I'd like to leave go get ./... out of this debate. I want everyone to remain laser focused on the usability issue of go test ./... matching vendored code.

There are certainly other issues with the Go 1.6+ behaviour, but for the sake of clarity, can those debates be taken elsewhere.

Thanks

@mmatczuk mmatczuk referenced this issue Feb 16, 2017

Merged

gofmt -s -w #384

@dcheney-atlassian

This comment has been minimized.

Copy link

dcheney-atlassian commented Mar 30, 2017

The cross platform argument is a red herring. The workaround doesn't have to function on every system under the sun. It only has to work in the environment it is applied to, be that a developer's shell or that build/test script you're writing.

But it does have to work on Windows, which are the largest population of Go users, and have neither shell aliases or nice things like grep, or substitution.

@slrz

This comment has been minimized.

Copy link

slrz commented Mar 30, 2017

This hasn't been true for over a decade. Any supported Windows variant includes a shell that, while somewhat different from Unix shells, is very well capable of such things. I wouldn't be surprised if it even provides a grep command by default. It certainly does so for many other common Unix programs (generally as an alias for the native Powershell command with roughly equivalent functionality).

@sgen

This comment has been minimized.

Copy link

sgen commented Mar 30, 2017

@slrz So with your suggestion windows users will now have to install a shell they're not familiar with, learn that shell, rewrite their scripts in it, and this is easier than you running go test ./... ./vendor/... until a complete vendoring solution is found?

@dlsniper

This comment has been minimized.

Copy link
Contributor

dlsniper commented Mar 30, 2017

@slrz if you could find something simpler than:

for /f "" %G in ('go list ./... ^| find /i /v "/vendor/"') do @go test %G

I'd be happy to hear about it.

Meanwhile, this feature was requested by the Go users, which proved with a fairly good technical evidence that not matching vendor/ in ... is desired or at least have a way to disable that behavior.

I believe that the current way that's implemented offers a good balance of that while still allowing vendor/ to be matched if so needed.

It will rarely be so that you need to type: ./... ./vendor/... ./.../vendor/... let alone ./... ./vendor/... ./.../vendor/... ./.../vendor/.../vendor/... (vendor flattening should be a thing by now in any package manager).

Most likely you'd need to type: ./... ./vendor/... which is still just one character longer than --no-vendor ./... and one shorter than --skip-vendor ./... but it's definitely easier to type / remember than the current way to do things.

Also, special cases do exist today, afaik testdata is one such case (if not only).

So please, try to understand that there are some of us who are really happy to see this, and per the commend here: #19090 (comment) that number might actually be bigger than you think. So a quality of life improvement for so many people while still allowing for the current behavior to exist is a welcomed change imho.

I hope this helps seeing "the other side" as well.

@slrz

This comment has been minimized.

Copy link

slrz commented Mar 30, 2017

@sgen Heck, we're talking about programmers here. They will figure out something to exclude items containing a constant substring from a list.

Assuming they even want that particular quirky behaviour instead of the one you'd expect from reading the description of patterns in the help text, without the part about this special case.

@spenczar

This comment has been minimized.

Copy link
Contributor

spenczar commented Mar 30, 2017

@slrz this isn't the quirk. The quirk is that vendor is a special directory for the go build tool. This completes the implementation of the quirk.

We were in a bad state previously when vendor was incompletely implemented. We were in the worst position of all - a quirk that was inconsistent.

If purity doesn't convince you, practicality should: In my experience teaching Go and being a local expert in a large-ish company on Go tools, this was an almost universal stumbling block when people went to lint or test their code. Nobody knew to use go list and grep ahead of time, and most of them wrinkled their nose and made some disparaging comment about go's tools.

@cznic

This comment has been minimized.

Copy link
Contributor

cznic commented Mar 30, 2017

@dcheney-atlassian

But it does have to work on Windows, which are the largest population of Go users, and have neither shell aliases or nice things like grep, or substitution.

Can you please cite some source of Windows Go users being "the largest population of Go users"?

Meanwhile, let me refer to a survey of 3,595 respondents:

When asked which operating systems they develop Go on, 63% of respondents say they use Linux, 44% use MacOS, and 19% use Windows, with multiple choices allowed and 49% of respondents developing on multiple systems. The 51% of responses choosing a single system split into 29% on Linux, 17% on MacOS, 5% on Windows, and 0.2% on other systems..

@dcheney-atlassian

This comment has been minimized.

Copy link

dcheney-atlassian commented Mar 30, 2017

The windows version is the most popular download of windows/dl. This was reported to me in private correspondence a year or so ago, I don't have access to verify this fact directly.

@dlsniper

This comment has been minimized.

Copy link
Contributor

dlsniper commented Mar 31, 2017

Out of respect for everyone, I think we should stop the bike shedding now and maybe open up an issue if we find a problem in the current implementation as asked by @rsc.

Finally, please note that comments like Heck, we're talking about programmers here. are really not in the spirit of cooperation / Go. Yes, we are all programmers, but that doesn't mean we all share the same opinion.

The fact that vendor/ is special is undeniable and if users really really care about traversing that, then maybe they should have to use special scripts instead of those who don't (I've borrowed this last bit from someone who said it in a private discussion).

Lets not have the team lock down the discussion as I consider that to be a bit unfriendly. Thank you.

@sgen

This comment has been minimized.

Copy link

sgen commented Mar 31, 2017

But I want to paint it chartreuse!

@slrz

This comment has been minimized.

Copy link

slrz commented Mar 31, 2017

@dlsniper

if you could find something simpler than:
for /f "" %G in ('go list ./... ^| find /i /v "/vendor/"') do @go test %G
I'd be happy to hear about it.

The following should work in a default Windows system, having installed only the Go MSI package:

go test $(go list ./... | sls -NotMatch '/vendor/')

The sls ("select string") seems to be the Powershell equivalent to grep.

lparth added a commit to lparth/go that referenced this issue Apr 13, 2017

cmd/go: exclude vendored packages from ... matches
By overwhelming popular demand, exclude vendored packages from ... matches,
by making ... never match the "vendor" element above a vendored package.

go help packages now reads:

    An import path is a pattern if it includes one or more "..." wildcards,
    each of which can match any string, including the empty string and
    strings containing slashes.  Such a pattern expands to all package
    directories found in the GOPATH trees with names matching the
    patterns.

    To make common patterns more convenient, there are two special cases.
    First, /... at the end of the pattern can match an empty string,
    so that net/... matches both net and packages in its subdirectories, like net/http.
    Second, any slash-separted pattern element containing a wildcard never
    participates in a match of the "vendor" element in the path of a vendored
    package, so that ./... does not match packages in subdirectories of
    ./vendor or ./mycode/vendor, but ./vendor/... and ./mycode/vendor/... do.
    Note, however, that a directory named vendor that itself contains code
    is not a vendored package: cmd/vendor would be a command named vendor,
    and the pattern cmd/... matches it.

Fixes golang#19090.

Change-Id: I985bf9571100da316c19fbfd19bb1e534a3c9e5f
Reviewed-on: https://go-review.googlesource.com/38745
Reviewed-by: Alan Donovan <adonovan@google.com>
@AhmetS

This comment has been minimized.

Copy link

AhmetS commented Apr 24, 2017

My soln till this mess gets sorted is with npm package.json. Something like the following:

{
  "name": "myapp",
  "version": "1.0.0",
  "scripts": {
    "test:win": "for /f \"\" %G in ('go list ./... ^| find /i /v \"/vendor/\"') do @go test -v %G",
    "test:unix": "go test -v $(go list ./... | grep -v /vendor/)",

    "bench:unix": "go test -bench=. -v $(go list ./... | grep -v /vendor/)",
    "bench:win": "for /f \"\" %G in ('go list ./... ^| find /i /v \"/vendor/\"') do @go test -v -bench=. %G",

    "start:unix": "go run *.go",
    "start:win": "for /f \"\" %G in ('dir /b *.go') do @go run %G"
  }
}

To test on windows npm run test:win, to bench on windows npm run bench:win, and to run on windows npm start:win

To test on linux/osx npm run test:unix, to bench on linux/osx npm run bench:unix, and to run on linux/osx npm start:unix

@mantzas

This comment has been minimized.

Copy link

mantzas commented May 19, 2017

better late than never!

keymon added a commit to redfactorlabs/concourse-smuggler-resource that referenced this issue Sep 25, 2017

Update all deps and add vendor directory
We update all the vendored dependencies and the golang version
by running:

         go get golang.org/x/sys/unix # Explicit because not added by mac
         godep get $(GO_PACKAGES)
         godep update -goversion
         godep update ./...
         godep save

We add the resulting vendor directory to the git repo, so we do not
need call godep anymore when calling the go commands[1]. In the
Makefile we add logic to ignore the ./vendor directory as
suggested in[2]

[1] https://golang.org/cmd/go/#hdr-Vendor_Directories
[2] golang/go#19090 (comment)

@golang golang locked and limited conversation to collaborators May 19, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.