Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: x/tools/go/packages: add no-network package lookup #40053

Open
james-lawrence opened this issue Jul 5, 2020 · 7 comments
Open

proposal: x/tools/go/packages: add no-network package lookup #40053

james-lawrence opened this issue Jul 5, 2020 · 7 comments

Comments

@james-lawrence
Copy link
Contributor

@james-lawrence james-lawrence commented Jul 5, 2020

I have a couple projects where I want to resolve the location of a package if it exists, but not retrieve/verify the package.

My initial attempts to use go/build.Context.Import trigger a retrieval of the package if its not found. which is undesirable due to both security and performance concerns.

I have no problem with that behavior, but go build and go install do not require network access and I'd prefer to not reinvent the wheel to locate packages that golang uses itself.

go/importer looked promising, but it suffers from the same issue as it uses go/build under the hood and it does not return the same information.

the underlying cause seems to be where Import shells out to go list to resolve the paths, though I stopped following the code once I hit the cmd execution in go/build.

Not opposed to implementing the changes myself, just looking for ideas/buyin/alternatives.

possible implementation ideas:

  1. add a Locate method to build.Context, same signature as Import but doesn't generate any network calls to resolve the package.
  2. add a NoNetwork to build.ImportMode flags. (obviously happy to bikeshed names)

What version of Go are you using (go version)?

go version go1.14.4 linux/amd64

Does this issue reproduce with the latest release?

yes

What did you do?

# clear out cache/gopath
mv ~/go{,.bak}
package main

import (
	"go/build"
	"log"
)

func main() {
	b := build.Default
	p, err := b.Import("github.com/lib/pq", "", build.FindOnly)
	if err != nil {
		log.Fatal(err)
	}

	log.Println("package found:", p.Name, p.Dir)
}

What did you expect to see?

an error about package not found.

What did you see instead?

the package was downloaded and found.

@gopherbot gopherbot added this to the Proposal milestone Jul 5, 2020
@rsc rsc added this to Incoming in Proposals Jul 8, 2020
@rsc
Copy link
Contributor

@rsc rsc commented Jul 8, 2020

golang.org/x/tools/go/packages is the right one to be using, not go/build.
That said, I don't think it handles this either.
But it probably could if we could figure out the go command flag to mean "don't use the network".

/cc @bcmills @matloob @jayconrod @ianthehat

@rsc rsc changed the title proposal: expose the ability to locate a package without accessing the network. proposal: x/tools/go/packages: add no-network package lookup Jul 8, 2020
@rsc rsc moved this from Incoming to Active in Proposals Jul 8, 2020
@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jul 8, 2020

I think this should be a feature of go list, for example, something like go list -f '{{.Dir}}' example.com/pkg could work in more cases. Both go/build and golang.org/x/tools/go/packages use go list to load package metadata, so if go list works, they would work.

GOPROXY is already used to control where modules are fetched from. GOPROXY=off is pretty close already. The main limitation is that you can't find the "latest" version of a module (or match any version query) when using GOPROXY=off. #32337 suggests letting go list -m -versions list cached versions when offline. That would get us most of the way there.

#26717 is also a request for a general mechanism for exploring the module cache.

@james-lawrence
Copy link
Contributor Author

@james-lawrence james-lawrence commented Jul 9, 2020

@rsc thanks for pointing me in the right direction.

@jayconrod go list is likely the right spot to implement a change. let me know if you need anything for this, including implementation work.

@rsc
Copy link
Contributor

@rsc rsc commented Jul 22, 2020

Sounds like we need to decide how to tell the go command to do this, and then circle back to x/tools/go/packages.

@rsc
Copy link
Contributor

@rsc rsc commented Jul 23, 2020

I talked to @jayconrod, @matloob, and @bcmills. The plan we suggest is to make GOPROXY=modcache mean use the module cache as the full proxy content, including for list requests, and then users would set that environment variable in the go/packages config.

@rsc
Copy link
Contributor

@rsc rsc commented Aug 5, 2020

I talked to @jayconrod, @matloob, and @bcmills again, and I also started a discussion on golang-tools about the right defaults for the go command.

At this point, it seems like a likely outcome could be that the go command stops doing network lookups during go list by default, which would then make this proposal unnecessary - x/tools/go/packages would stop doing network lookups too.

So we should probably wait until that gets resolved. Putting this proposal on hold.

@james-lawrence
Copy link
Contributor Author

@james-lawrence james-lawrence commented Mar 7, 2021

as a related note: part of the problem here is the lack of a well known API for interacting with dependency source code. (downloading, package directory locations/files) I had thought go/build was that but @rsc corrected me here. something to chew on for thought.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants