vendor: no docs on how to update a module in std

[bradfitz]

I've been flailing around all morning fighting the new system of updating modules in std.

I want to update x/net in std because the golang.org/x/net/http/httpproxy we have in std is old and doesn't support ALL_PROXY.

Note that all_proxy is present in golang.org/x/net:

bradfitz@go:~/src/golang.org/x/net$ git show-ref HEAD
9ce7a6920f093fc0b908c4a5f66ae049110f417e refs/remotes/origin/HEAD
bradfitz@go:~/src/golang.org/x/net$ git grep -i all_proxy
proxy/proxy.go:         names: []string{"ALL_PROXY", "all_proxy"},
proxy/proxy_test.go:            fmt.Fprintf(&buf, "all_proxy=%q", t.allProxyEnv)
proxy/proxy_test.go:            os.Setenv("ALL_PROXY", tt.allProxyEnv)

But it's not present in std:

bradfitz@go:~$ cd $GOROOT/src/
bradfitz@go:~/go/src$ git grep -i all_proxy

So, let's update it...

bradfitz@go:~/go/src$ go get -u golang.org/x/net/...@latest
go: finding golang.org/x/net latest
go: finding golang.org/x/sys latest
go: finding golang.org/x/crypto latest
bradfitz@go:~/go/src$ git diff go.mod                                                                                                                                                                       diff --git a/src/go.mod b/src/go.mod
index a527f9a244..047e43f864 100644
--- a/src/go.mod
+++ b/src/go.mod
@@ -3,9 +3,9 @@ module std
 go 1.12
 
 require (
-       golang.org/x/crypto v0.0.0-20190424203555-c05e17bb3b2d
-       golang.org/x/net v0.0.0-20190424112056-4829fb13d2c6
-       golang.org/x/sys v0.0.0-20190425145619-16072639606e // indirect
+       golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734
+       golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09
+       golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd // indirect
        golang.org/x/text v0.3.2 // indirect
-       golang.org/x/tools v0.0.0-20190425214124-2d660fb8a000 // indirect
+       golang.org/x/tools v0.0.0-20190501045030-23463209683d // indirect
 )

And go mod vendor perhaps?

bradfitz@go:~/go/src$ go mod vendor
bradfitz@go:~/go/src$ git grep -i all_proxy

Nope. Still not present.

What changed?

bradfitz@go:~/go/src$ git diff --stat
 src/go.mod                                                |   8 ++---
 src/go.sum                                                |   7 ++++
 src/vendor/golang.org/x/sys/unix/mkall.sh                 |  14 +++++---
 src/vendor/golang.org/x/sys/unix/mksysctl_openbsd.pl      | 265 ----------------------------------------------------------------------------------------------------------------------------------------------------
 src/vendor/golang.org/x/sys/unix/sockcmsg_unix.go         |   8 ++---
 src/vendor/golang.org/x/sys/unix/syscall.go               |   1 -
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_386.go   |   2 ++
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_amd64.go |   2 +-
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_arm.go   |   4 ++-
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_arm64.go |   2 +-
 src/vendor/modules.txt                                    |  14 ++++----
 11 files changed, 39 insertions(+), 288 deletions(-)

... other stuff changed, but not golang.org/x/net/...

I tried to get get a specific version rather than "latest"

bradfitz@go:~/go/src$ go get -u golang.org/x/net/http/httpproxy@v0.0.0-20190501004415-9ce7a6920f09
go: finding golang.org/x/net/http/httpproxy v0.0.0-20190501004415-9ce7a6920f09
go: finding golang.org/x/net/http v0.0.0-20190501004415-9ce7a6920f09
bradfitz@go:~/go/src$ go mod vendor
bradfitz@go:~/go/src$ git grep -i all_proxy
bradfitz@go:~/go/src$ git di --stat
 src/go.mod                                                |   8 ++---
 src/go.sum                                                |   7 ++++
 src/vendor/golang.org/x/sys/unix/mkall.sh                 |  14 +++++---
 src/vendor/golang.org/x/sys/unix/mksysctl_openbsd.pl      | 265 ----------------------------------------------------------------------------------------------------------------------------------------------------
 src/vendor/golang.org/x/sys/unix/sockcmsg_unix.go         |   8 ++---
 src/vendor/golang.org/x/sys/unix/syscall.go               |   1 -
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_386.go   |   2 ++
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_amd64.go |   2 +-
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_arm.go   |   4 ++-
 src/vendor/golang.org/x/sys/unix/zsysctl_openbsd_arm64.go |   2 +-
 src/vendor/modules.txt                                    |  14 ++++----
 11 files changed, 39 insertions(+), 288 deletions(-)

Still no good.

A README.txt would really help.

Also, what is vendor/modules.txt? What role does it play and is it hand maintained or tool maintained and where is it documented? Does it support comments? Could it have some comments saying what it is?

Please help. (and document)

[ianlancetaylor]

This is the thread that discussed this: https://groups.google.com/d/msg/golang-dev/xcVJDj5GJ84/6XF1X8yhBwAJ .

(Austin asked for docs there, but I don't know if any were written.)

CC @bcmills

[bradfitz]

I also tried go get -m. Same results.

[bradfitz]

This is the thread that discussed this:

I'm still not able to make it work.

[thepudds]

Maybe this is "expected" for some reason, or maybe there is something specific to how GOROOT is set, or some other stdlib-specific step, but if this was a normal module (rather than stdlib), then this would seem to be a bug.

In this case, it looks like running go mod vendor does update src/vendor/modules.txt with the proper pseudoversion for golang.org/x/net (including expected commit of 9ce7a6920f09):

# golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09
golang.org/x/net/dns/dnsmessage
golang.org/x/net/lif
golang.org/x/net/route
golang.org/x/net/http/httpguts
golang.org/x/net/http/httpproxy
golang.org/x/net/http2/hpack
golang.org/x/net/idna
golang.org/x/net/nettest

...but the corresponding .go source files in vendor are not.

Also, what is vendor/modules.txt? What role does it play and is it hand maintained or tool maintained and where is it documented?

vendor/modules.txt is maintained by the go tool, and is not normally hand edited. I believe the format is consider an implementation detail and not documented externally... but it often (always?) matches the output of go mod vendor -v, which is lightly documented:

usage: go mod vendor [-v]

Vendor resets the main module's vendor directory to include all packages
needed to build and test all the main module's packages.
It does not include test code for vendored packages.

The -v flag causes vendor to print the names of vendored
modules and packages to standard error.

In this case, go mod vendor -v does match the contents of ./vendor/modules.txt.

[bradfitz]

Okay, @jayconrod points out that I'm confused. I thought the httpproxy package (which we use) was the same or depended on the golang.org/x/net/proxy package (which contains the all_proxy stuff), but it doesn't. They're separate. I think that transition never completed.

But docs would still be nice.

[thepudds]

Two other quick comments.

It seems go mod vendor -v includes the package golang.org/x/net/http/httpproxy in its output, but not the package golang.org/x/net/proxy

The module cache in GOPATH/pkg/mod does seem to end up with golang.org/x/net/proxy with all_proxy:

$ cd $(go list -f '{{.Dir}}' -m golang.org/x/net)
$ pwd
.../go/pkg/mod/golang.org/x/net@v0.0.0-20190502183928-7f726cade0ab
$ grep -ri all_proxy
proxy/dial_test.go:             if err = os.Setenv("ALL_PROXY", fmt.Sprintf("socks5://%s", s.Addr().String())); 
...

I have never tried to vendor std before, so it could be I am making a simple mistake.

edit: ok, seems like mystery is now solved.

[jayconrod]

So it seems vendoring is working as intended; I was initially confused because I misinterpreted bisect results during a meeting.

Documentation definitely needed. I thought there was a README at some point, but I'm unable to find it in the history.

[rsc]

We'll add the docs. I also have a pending CL that makes essentially all go commands after 'go get whatever@latest' die with a message saying 'now you run go mod vendor'.

[gopherbot]

Change https://golang.org/cl/174999 mentions this issue: all: document vendoring in the standard library