Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: `link` reports "symbol listed multiple times" if go:linkname into both vendor package and non-vendor package #42401

Closed
chenzhuoyu opened this issue Nov 5, 2020 · 7 comments

Comments

@chenzhuoyu
Copy link

@chenzhuoyu chenzhuoyu commented Nov 5, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/myusername/Library/Caches/go-build"
GOENV="/Users/myusername/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/myusername/GolangProjects/pkg/mod"
GONOPROXY="*.mycompanydomain.org"
GONOSUMDB="*.mycompanydomain.org"
GOOS="darwin"
GOPATH="/Users/myusername/GolangProjects"
GOPRIVATE="*.mycompanydomain.org"
GOPROXY="https://go-mod-proxy.mycompanydomain.org,https://goproxy.cn,https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15.3/libexec"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/myusername/Sources/tests/test_go/main/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/h3/c5bgpd9921d0tbkpf5n9vh2w0000gn/T/go-build518026853=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  1. Create a directory structure like this:
.
├── foo
│   ├── foo.go
│   └── go.mod
└── main
    ├── go.mod
    └── main.go
// $ cat foo/go.mod 
module github.com/chenzhuoyu/env

go 1.15
// $ cat foo/foo.go 
package foo

var bar string

func GetBar() string {
    return bar
}

func init() {
    bar = ""
}
// $ cat main/go.mod 
module github.com/chenzhuoyu/linker_issue

go 1.15

replace github.com/chenzhuoyu/foo v0.0.0 => ../foo

require github.com/chenzhuoyu/foo v0.0.0
// $ cat main/main.go 
package main

import (
    _ `unsafe`

    `github.com/chenzhuoyu/foo`
)

//go:linkname v1 github.com/chenzhuoyu/foo.bar
var v1 string

//go:linkname v2 github.com/chenzhuoyu/linkname/vendor/github.com/chenzhuoyu/foo.bar
var v2 string

func main() {
    v1 = v2
    foo.GetBar()
}
  1. Build the main package.
$ cd main
$ go build .

What did you expect to see?

The main package builds just fine.

What did you see instead?

$ go build .
# github.com/chenzhuoyu/linker_issue
2020/11/05 23:56:41 symbol go.info.github.com/chenzhuoyu/foo.bar listed multiple times

I have also tried:

  • go 1.14.10 works fine.
$ cd main
$ ls
go.mod  main.go
$ /usr/local/opt/go@1.14/bin/go version
go version go1.14.10 darwin/amd64
$ /usr/local/opt/go@1.14/bin/go build .
$ ls
go.mod  linker_issue*  main.go
  • Passing -ldflags='-w' to the compiler also works fine.
$ go build -ldflags='-w' .
$ ls
go.mod  linker_issue*  main.go
  • After searching for several hours, I found these related issues, but they are not very helpful to me.

#11699
#15926
#28159

@thanm thanm added this to the Go1.16 milestone Nov 5, 2020
@thanm
Copy link
Member

@thanm thanm commented Nov 5, 2020

I'll take a look.

@thanm
Copy link
Member

@thanm thanm commented Nov 5, 2020

A note before getting into the details of the bug: please be aware that the behavior + implementation of "//go:linkname" is explicitly not covered by the Go 1 compatibility guarantee; the fact that importing "unsafe" is required is intended to reinforce this.

As the toolchain changes over time, the implementation and behavior of "//go:linkname" can evolve. There are uses of "//go:linkname" that may work with version X of the Go toolchain and then not work with version X+1. For example, using "//go:linkname" to reach from package A into a dependent package B to access one of B's non-exported symbols is something that works today, but may very well not work with future versions of Go.

Regarding the Go program in the bug:

This program contains behavior that happened to work with the Go 1.14 toolchain implementation of "//go:linkname" but does not work with Go 1.15. In Go 1.14, linking and symbol resolution is done using an entirely name-driven basis; all (or virtuall all) symbols are named, and at link time two symbols with the same name wind up mapping to same linker symbol, regardless of which package or compilation unit the symbol was defined in.

What this means is that for Go 1.14 when you write

  //go:linkname v1 github.com/chenzhuoyu/foo.bar
  var v1 string

this has the effect of making "v1" (defined in main.go) and "bar" (defined in foo.go) virtually the same symbol at link time. If you look in the final binary produced by Go 1.14 for this example, there are only two "foo.bar" variables.

In Go 1.15 we moved to a new object file format and a new way of doing symbol resolution. For most regular Go symbol definitions (variables, functions) the compiler assigns the symbol an index within the object file; when the linker reads the object file, it sees the symbol (roughly) as a tuple <P,S> where P is a numeric package ID and S is a numeric symbol ID.

What this means for Go 1.15 is that there are actually three "foo.bar" strings: one defined in package foo and two in package main; each has a unique identity using the new index-based scheme. Two of the three symbols happen to have the same name, which results in a collision during DWARF generation.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 5, 2020

I'm not sure I understand the issue.

github.com/chenzhuoyu/linkname/vendor/github.com/chenzhuoyu/foo.bar

Where do you vendor this? I didn't see any vendor directory or any vendor command.

Also, do you run into this issue in real code, or just this synthetic example?

@chenzhuoyu
Copy link
Author

@chenzhuoyu chenzhuoyu commented Nov 6, 2020

Where do you vendor this? I didn't see any vendor directory or any vendor command.

I don't, I just write it there.

Also, do you run into this issue in real code, or just this synthetic example?

I ran into this issue in real code in our library, it is only used for unittests, so there are a lot of "hacks" in it. The code above is just a simplification.

Some of our legacy projects use dep to manage dependencies, so the code needs to work with both vendored projects and non-vendored projects, but upgrading to Go 1.15.3 breaks a lot of our tests. :(

Maybe we should consider refactoring the library and get rid of go:linkname, but that will be a lot of work. :(

@chenzhuoyu
Copy link
Author

@chenzhuoyu chenzhuoyu commented Nov 6, 2020

A note before getting into the details of the bug: please be aware that the behavior + implementation of "//go:linkname" is explicitly not covered by the Go 1 compatibility guarantee; the fact that importing "unsafe" is required is intended to reinforce this.

We only use "//go:linkname" for unittests, not in our production code.

There are some not-so-well-designed libraries that come with a lot of side-effects (performing I/O within init, reading environment variables, etc), which can cause our tests to be very unstable.

The "//go:linkname" is a way we found to "hack" these libraries and set them into a certain state for our tests.

As the toolchain changes over time, the implementation and behavior of "//go:linkname" can evolve. There are uses of "//go:linkname" that may work with version X of the Go toolchain and then not work with version X+1. For example, using "//go:linkname" to reach from package A into a dependent package B to access one of B's non-exported symbols is something that works today, but may very well not work with future versions of Go.

Should I consider "//go:linkname" as some sort of "undefined behavior" ?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 6, 2020

I wouldn't say that //go:linkname is undefined behavior. But it is unsafe, and it is not covered by the Go 1 compatibility guarantee. If you choose to use it, you must be prepared to adjust your use with each new Go release.

In general I would advise against using it.

@thanm thanm removed this from the Go1.16 milestone Nov 6, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 7, 2020

Change https://golang.org/cl/268178 mentions this issue: cmd/compile: make sure linkname'd symbol is non-package

@gopherbot gopherbot closed this in a444458 Nov 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.