Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add a way to obtain a package's build ID via list -json #37281

Closed
mvdan opened this issue Feb 18, 2020 · 3 comments
Closed

cmd/go: add a way to obtain a package's build ID via list -json #37281

mvdan opened this issue Feb 18, 2020 · 3 comments

Comments

@mvdan
Copy link
Member

@mvdan mvdan commented Feb 18, 2020

The toolchain computes build IDs of packages from their build input - which I believe includes a package's source code, plus the build IDs of all their dependencies.

This can be seen if one runs go build -x, as compiler invocations contain -buildid $BUILD_ID. Similarly, a package's output file contains such build ID as well, so we can obtain it via these two commands:

$ go list -f {{.Export}} -export os
/home/mvdan/tip/pkg/linux_amd64/os.a
$ go tool buildid $(go list -f {{.Export}} -export os)
WCQPVNh2igTNOQyg4V2a/aDZysAb7fmq4QDTG8hDz

This is rather unfortunate for tools that need to look at thousands of packages at a time, though. One could use the build ID to see if a package hasn't had any changes, and thus avoid extra work. Even if we have thousands of packages, we can load them all at once with one single go list -json call - however, we will need one extra exec call to go tool buildid per package, whose cost can add up quickly.

I propose that we add a new field to go list -json to expose said build ID. For example, BuildID string. It would be fine if populating this field required a flag like -export.

I also realise that build IDs are an internal toolchain detail, which should only matter to compiler and toolchain authors. However, there are plenty of third-party tools out there which are designed to be closely coupled with the toolchain or compiler. For example, tools designed to run via go build -toolexec=mytool can use build IDs to avoid work, or to produce deterministic output based on the input. This is exactly the kind of thing I do for a code obfuscator, only with lots of go tool buildid calls.

I also am not asking for the build ID format to be documented or stabilized. A tool shouldn't interpret the string as more than a hash if it wants to remain compatible with past and future Go versions. Or, if it really wants to parse the hash, it should only do so with very specific versions of the Go tool.

/cc @dominikh

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Feb 18, 2020

@bcmills
Copy link
Member

@bcmills bcmills commented May 13, 2020

Let's see how invasive the change is, and how clean the workflow ends up (as demonstrated in an integration test).

@bcmills bcmills added the help wanted label May 13, 2020
@mvdan mvdan self-assigned this May 13, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 19, 2020

Change https://golang.org/cl/263542 mentions this issue: cmd/go: add BuildID to list -json -export

@gopherbot gopherbot closed this in 0bf507e Oct 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.