Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: generating C header files for transitive dependencies is awkward #35715

Open
bcmills opened this issue Nov 20, 2019 · 6 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 20, 2019

As discovered in CL 208117 (#30316), cmd/go (as of Go 1.14) lacks a good way to generate C header files for Go libraries in order to use them from C.

go install -buildmode=c-archive and go install -buildmode=c-shared will, as a side-effect, generate header files for the requested package. However, -buildmode=c-archive and -buildmode=c-shared today require the named package to be a package main, and those commands do not generate headers for the (transitive) Go dependencies of the named package.

The -i flag does generate headers for transitive dependencies, but it also has a second, unwanted side effect: it builds (and attempts to install) libraries for the transitive dependencies of the named package, including transitive dependencies in the standard library. That fails if the user cannot write to GOROOT/pkg, such as when a non-root user is working with a go tool installed as root.

As a workaround, one can run go tool cgo -exportheader, but go tool cgo accepts a list of files rather than a list of packages, and generates a number of additional outputs (normally in an _obj subdirectory) that the C caller does not directly need. So we're still doing extra work, just in a different location.

We should see what we can do to make it easier to generate C headers without unnecessary or unwanted side-effects.

CC @ianlancetaylor @jayconrod

@bcmills bcmills added this to the Backlog milestone Nov 20, 2019
@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Nov 20, 2019

One possibility might be to add another -buildmode, such as -buildmode=c-header, that does whatever work is needed (and possibly writes extra artifacts into GOCACHE) but only emits (or installs) header files rather than entire archives.

That could be used as a more direct substitute for the go tool cgo call with specific package arguments.

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Nov 20, 2019

Another option might be to make the -i flag suppress errors for standard-library targets, so that the go install -i command would work even if GOROOT is not writable.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 20, 2019

Change https://golang.org/cl/208117 mentions this issue: misc/cgo/testcarchive: avoid writing to GOROOT in tests

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 20, 2019

Change https://golang.org/cl/208119 mentions this issue: misc/cgo/testcshared: avoid writing to GOROOT in tests

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 22, 2019

We could have go install install all headers for imported packages, even without -i. That would be a change in behavior, but only for packages that are not main but have exported functions, which is an uncommon case.

@bcmills

This comment has been minimized.

Copy link
Member Author

@bcmills bcmills commented Nov 22, 2019

Maybe, but the import itself seems like a bit of a non-sequitur: if the C code is using identifiers from package a, then it seems like the build process should mention package a rather than some other package b that happens to import a.

On the other hand, I suppose that the C library must link against some (shared or archive) file generated from a main package, so having go install generate all of the headers for identifiers linked into that main package doesn't seem terrible.

gopherbot pushed a commit that referenced this issue Nov 22, 2019
Also add a -testwork flag to facilitate debugging the test itself.

Three of the tests of this package invoked 'go install -i
-buildmode=c-archive' in order to generate an archive as well as
multiple C header files.

Unfortunately, the behavior of the '-i' flag is inappropriately broad
for this use-case: it not only generates the library and header files
(as desired), but also attempts to install a number of (unnecessary)
archive files for transitive dependencies to
GOROOT/pkg/$GOOS_$GOARCH_shared, which may not be writable — for
example, if GOROOT is owned by the root user but the test is being run
by a non-root user.

Instead, for now we generate the header files for transitive dependencies
separately by running 'go tool cgo -exportheader'.

In the future, we should consider how to improve the ergonomics for
generating transitive header files without coupling that to
unnecessary library installation.

Updates #28387
Updates #30316
Updates #35715

Change-Id: I3d483f84e22058561efe740aa4885fc3f26137b5
Reviewed-on: https://go-review.googlesource.com/c/go/+/208117
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue Nov 22, 2019
The tests in this package invoked 'go install -i -buildmode=c-shared'
in order to generate an archive as well as multiple C header files.

Unfortunately, the behavior of the '-i' flag is inappropriately broad
for this use-case: it not only generates the library and header files
(as desired), but also attempts to install a number of (unnecessary)
archive files for transitive dependencies to
GOROOT/pkg/$GOOS_$GOARCH_testcshared_shared, which may not be writable
— for example, if GOROOT is owned by the root user but the test is
being run by a non-root user.

Instead, for now we generate the header files for transitive dependencies
separately by running 'go tool cgo -exportheader'.

In the future, we should consider how to improve the ergonomics for
generating transitive header files without coupling that to
unnecessary library installation.

Updates #28387
Updates #30316
Updates #35715

Change-Id: I622426a860828020d98f7040636f374e5c766d28
Reviewed-on: https://go-review.googlesource.com/c/go/+/208119
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.