cmd/go: cache meta lookups during go get

[bradfitz]

Installing godoc, I see many meta tag lookups. Can we do better?

$ go get -u -v golang.org/x/tools/cmd/godoc
Fetching https://golang.org/x/tools/cmd/godoc?go-get=1
Parsing meta tags from https://golang.org/x/tools/cmd/godoc?go-get=1 (status code 200)
get "golang.org/x/tools/cmd/godoc": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/cmd/godoc?go-get=1
get "golang.org/x/tools/cmd/godoc": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
golang.org/x/tools (download)
Fetching https://golang.org/x/tools/astutil?go-get=1
Parsing meta tags from https://golang.org/x/tools/astutil?go-get=1 (status code 200)
get "golang.org/x/tools/astutil": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/astutil?go-get=1
get "golang.org/x/tools/astutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/blog?go-get=1
Parsing meta tags from https://golang.org/x/tools/blog?go-get=1 (status code 200)
get "golang.org/x/tools/blog": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/blog?go-get=1
get "golang.org/x/tools/blog": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/blog/atom?go-get=1
Parsing meta tags from https://golang.org/x/tools/blog/atom?go-get=1 (status code 200)
get "golang.org/x/tools/blog/atom": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/blog/atom?go-get=1
get "golang.org/x/tools/blog/atom": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/present?go-get=1
Parsing meta tags from https://golang.org/x/tools/present?go-get=1 (status code 200)
get "golang.org/x/tools/present": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/present?go-get=1
get "golang.org/x/tools/present": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/container/intsets?go-get=1
Parsing meta tags from https://golang.org/x/tools/container/intsets?go-get=1 (status code 200)
get "golang.org/x/tools/container/intsets": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/container/intsets?go-get=1
get "golang.org/x/tools/container/intsets": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/callgraph?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/callgraph?go-get=1 (status code 200)
get "golang.org/x/tools/go/callgraph": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/callgraph?go-get=1
get "golang.org/x/tools/go/callgraph": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/exact?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/exact?go-get=1 (status code 200)
get "golang.org/x/tools/go/exact": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/exact?go-get=1
get "golang.org/x/tools/go/exact": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/gcimporter?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/gcimporter?go-get=1 (status code 200)
get "golang.org/x/tools/go/gcimporter": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/gcimporter?go-get=1
get "golang.org/x/tools/go/gcimporter": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/types?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/types?go-get=1 (status code 200)
get "golang.org/x/tools/go/types": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/types?go-get=1
get "golang.org/x/tools/go/types": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/loader?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/loader?go-get=1 (status code 200)
get "golang.org/x/tools/go/loader": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/loader?go-get=1
get "golang.org/x/tools/go/loader": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/ssa?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/ssa?go-get=1 (status code 200)
get "golang.org/x/tools/go/ssa": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/ssa?go-get=1
get "golang.org/x/tools/go/ssa": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/types/typeutil?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/types/typeutil?go-get=1 (status code 200)
get "golang.org/x/tools/go/types/typeutil": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/types/typeutil?go-get=1
get "golang.org/x/tools/go/types/typeutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/pointer?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/pointer?go-get=1 (status code 200)
get "golang.org/x/tools/go/pointer": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/pointer?go-get=1
get "golang.org/x/tools/go/pointer": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/go/ssa/ssautil?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/ssa/ssautil?go-get=1 (status code 200)
get "golang.org/x/tools/go/ssa/ssautil": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/ssa/ssautil?go-get=1
get "golang.org/x/tools/go/ssa/ssautil": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc?go-get=1 (status code 200)
get "golang.org/x/tools/godoc": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc?go-get=1
get "golang.org/x/tools/godoc": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/analysis?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/analysis?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/analysis": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/analysis?go-get=1
get "golang.org/x/tools/godoc/analysis": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/util?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/util?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/util": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/util?go-get=1
get "golang.org/x/tools/godoc/util": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/vfs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs?go-get=1
get "golang.org/x/tools/godoc/vfs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/vfs/httpfs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs/httpfs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs/httpfs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs/httpfs?go-get=1
get "golang.org/x/tools/godoc/vfs/httpfs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/redirect?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/redirect?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/redirect": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/redirect?go-get=1
get "golang.org/x/tools/godoc/redirect": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/static?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/static?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/static": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/static?go-get=1
get "golang.org/x/tools/godoc/static": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/vfs/gatefs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs/gatefs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs/gatefs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs/gatefs?go-get=1
get "golang.org/x/tools/godoc/vfs/gatefs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/vfs/mapfs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs/mapfs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs/mapfs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs/mapfs?go-get=1
get "golang.org/x/tools/godoc/vfs/mapfs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/godoc/vfs/zipfs?go-get=1
Parsing meta tags from https://golang.org/x/tools/godoc/vfs/zipfs?go-get=1 (status code 200)
get "golang.org/x/tools/godoc/vfs/zipfs": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/godoc/vfs/zipfs?go-get=1
get "golang.org/x/tools/godoc/vfs/zipfs": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
Fetching https://golang.org/x/tools/playground?go-get=1
Parsing meta tags from https://golang.org/x/tools/playground?go-get=1 (status code 200)
get "golang.org/x/tools/playground": found meta tag main.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/playground?go-get=1
get "golang.org/x/tools/playground": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
golang.org/x/tools/blog/atom
golang.org/x/tools/present
golang.org/x/tools/go/exact
golang.org/x/tools/astutil
golang.org/x/tools/container/intsets
golang.org/x/tools/godoc/vfs
golang.org/x/tools/go/types
golang.org/x/tools/godoc/util
golang.org/x/tools/godoc/vfs/httpfs
golang.org/x/tools/godoc/redirect
golang.org/x/tools/blog
golang.org/x/tools/godoc/static
golang.org/x/tools/godoc/vfs/gatefs
golang.org/x/tools/godoc/vfs/mapfs
golang.org/x/tools/godoc/vfs/zipfs
golang.org/x/tools/playground
golang.org/x/tools/go/gcimporter
golang.org/x/tools/go/types/typeutil
golang.org/x/tools/go/loader
golang.org/x/tools/go/ssa
golang.org/x/tools/go/callgraph
golang.org/x/tools/go/ssa/ssautil
golang.org/x/tools/go/pointer
golang.org/x/tools/godoc/analysis
golang.org/x/tools/godoc
golang.org/x/tools/cmd/godoc

[rsc]

It's clearly a mistake to keep checking the same URLs over and over again during a single 'go get'. I think you wrote the code Brad. :-)

[bradfitz]

Yup, will fix. I keep hitting this.

[cespare]

Ah, I guess this is the other facet to the issue I described in https://groups.google.com/d/msg/golang-nuts/ixF0h1YYCoQ/9wfBu4deaM0J. (That is, combine the meta lookups with https timeouts -- see #10120 -- and go get can take several minutes.)

[bradfitz]

Sent https://go-review.googlesource.com/8727