Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: sectionForAddress(0xA9D67F) address not in any section file #40974

Open
wkoszek opened this issue Aug 22, 2020 · 13 comments
Open

cmd/link: sectionForAddress(0xA9D67F) address not in any section file #40974

wkoszek opened this issue Aug 22, 2020 · 13 comments
Milestone

Comments

@wkoszek
Copy link

@wkoszek wkoszek commented Aug 22, 2020

What version of Go are you using (go version)?

go version
go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/wk/Library/Caches/go-build"
GOENV="/Users/wk/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/wk/gopath/pkg/mod"
GONOPROXY="gitlab.com/segmed"
GONOSUMDB="gitlab.com/segmed"
GOOS="darwin"
GOPATH="/Users/wk/gopath"
GOPRIVATE="gitlab.com/segmed"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/wk/r/segmed/dataops/cli/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-build586495030=/tmp/go-build -gno-record-gcc-switches -fno-common"
Darwin wkoszek_mba 19.5.0 Darwin Kernel Version 19.5.0: Thu Apr 30 07:42:45 PDT 2020; root:xnu-6153.121.1~3/RELEASE_X86_64 x86_64

What did you do?

I decided to upgrade today:

20:04:15][wk@wkoszek_mba:~/r/segmed/dataops/cli]$ brew upgrade golang
==> Upgrading 1 outdated package:
golang 1.14.6 -> 1.15
==> Upgrading golang 1.14.6 -> 1.15
==> Downloading https://homebrew.bintray.com/bottles/go-1.15.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/36ef2e4cc3ecc84e0c1e98580a63f8d417efc8831f0c59b0a7e13d517c61353b?response-content-disposition=attachment%3Bfilename%3D%22go-1.15.catalina.bottle.tar.gz%22&Poli
######################################################################## 100.0%
==> Pouring go-1.15.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/go/1.15: 9,769 files, 494.3MB
Removing: /usr/local/Cellar/go/1.14.6... (9,453 files, 424.7MB)
[20:05:29][wk@wkoszek_mba:~/r/segmed/dataops/cli]$ make

I upgraded only Golang, nothing else.

This resulted in my project crashing at building of test executable:

$ make t
../tools/pre-make.py
../tools/segmed_lint
gofmt -w *.go                       # formatting of Golang according to standard
goimports -w *.go                   # getting rid of too many imports/too few of them
gotags *.go > tags                  # tags for easy browsing
golint  .                           # linting (static analysis) of Go code
go test -o segmed.test -v -cover -c  # compile tests to the binary segmed.test. It doesn't run anything
# gitlab.com/segmed/dataops/cli.test
/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: sectionForAddress(0xA9D67F) address not in any section file '/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-link-819363190/go.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [segmed.test] Error 2

I parametrized my makefile and fetched 2 versions of Golang from the website:

  • go1.14.7.darwin-amd64.tar.gz
  • go1.15.darwin-amd64.tar.gz

Reproducing of 1.15 issue:

$ make GODIRPREFIX=./go1.15.darwin-amd64/go/bin/
../tools/pre-make.py
../tools/segmed_lint
./go1.15.darwin-amd64/go/bin/gofmt -w *.go                       # formatting of Golang according to standard
goimports -w *.go                   # getting rid of too many imports/too few of them
gotags *.go > tags                  # tags for easy browsing
golint  .                           # linting (static analysis) of Go code
go build -ldflags "-w -s" -o segmed *.go             # actually building stuff
GOARCH=amd64 GOOS=linux  go build -ldflags "-w -s" -o segmed.linux *.go
./go1.15.darwin-amd64/go/bin/go test -o segmed.test -v -cover -c  # compile tests to the binary segmed.test. It doesn't run anything
# gitlab.com/segmed/dataops/cli.test
/Users/wk/r/segmed/dataops/cli/go1.15.darwin-amd64/go/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: sectionForAddress(0xA9D67F) address not in any section file '/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-link-400480254/go.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [segmed.test] Error 2

Proof 1.14.7 is OK:

$ make GODIRPREFIX=./go1.14.7.darwin-amd64/bin/
../tools/pre-make.py
../tools/segmed_lint
./go1.14.7.darwin-amd64/bin/gofmt -w *.go                       # formatting of Golang according to standard
goimports -w *.go                   # getting rid of too many imports/too few of them
gotags *.go > tags                  # tags for easy browsing
golint  .                           # linting (static analysis) of Go code
go build -ldflags "-w -s" -o segmed *.go             # actually building stuff
GOARCH=amd64 GOOS=linux  go build -ldflags "-w -s" -o segmed.linux *.go
./go1.14.7.darwin-amd64/bin/go test -o segmed.test -v -cover -c  # compile tests to the binary segmed.test. It doesn't run anything
$

What did you expect to see?

I'd expect 1.15 to behave like 1.14.7

What did you see instead?

# gitlab.com/segmed/dataops/cli.test
/Users/wk/r/segmed/dataops/cli/go1.15.darwin-amd64/go/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: sectionForAddress(0xA9D67F) address not in any section file '/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-link-400480254/go.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

More findings

The structure of the project is dataops/{cli, dephi, debug}. In cli/ dir I have work-in-progress file asset.go which depends on our private dephi module published on GitLab. dephi module just carries 1.8MB of string constants which are meant to be compiled into the package. In dataops/dephi, I use pkger to get:

-rw-rw-r--    1 wk  staff  1855629 Aug 21 20:47 pkged.go

Other than that, "dephi" is very trivial -- 1 .go file -- which returns constants as []string. Now the funny part: cli/asset.go is very-work-in-progress: it has AssetsList() which isn't called from anywhere. But by moving asset.go out of the project, I made it compile fine with go1.15.

To isolate whether asset.go is a culprit I made debug/ directory, with some mocks, but I also depend on asset.go and I use dephi. It works OK:

$ make
../tools/pre-make.py
../tools/segmed_lint
gofmt -w *.go                       # formatting of Golang according to standard
goimports -w *.go                   # getting rid of too many imports/too few of them
gotags *.go > tags                  # tags for easy browsing
golint  .                           # linting (static analysis) of Go code
config.go:3:6: exported type Config should have comment or be unexported
main.go:5:5: exported var GlobalVar should have comment or be unexported
go build -ldflags "-w -s" -o segmed *.go             # actually building stuff
GOARCH=amd64 GOOS=linux  go build -ldflags "-w -s" -o segmed.linux *.go
go test -o segmed.test -v -cover -c  # compile tests to the binary segmed.test. It doesn't run anything
[21:57:28][wk@wkoszek_mba:~/r/segmed/dataops/debug]$ ls -la
total 19380
drwxrwxr-x  15 wk  staff      480 Aug 21 21:57 .
drwxrwxr-x  46 wk  staff     1472 Aug 21 21:52 ..
-rw-rw-r--   1 wk  staff      533 Aug 21 21:39 asset.go
-rw-rw-r--   1 wk  staff       37 Aug 21 21:40 config.go
-rw-rw-r--   1 wk  staff      438 Aug 21 21:41 go.mod
-rw-rw-r--   1 wk  staff     5711 Aug 21 21:40 go.sum
-rw-rw-r--   1 wk  staff      107 Aug 21 21:41 helper.go
-rw-rw-r--   1 wk  staff      116 Aug 21 21:42 main.go
-rw-rw-r--   1 wk  staff       85 Aug 21 21:37 main_test.go
-rw-rw-r--   1 wk  staff     2763 Aug 21 21:35 makefile
-rwxrwxr-x   1 wk  staff  6237020 Aug 21 21:57 segmed
-rwxrwxr-x   1 wk  staff  5578752 Aug 21 21:57 segmed.linux
-rwxrwxr-x   1 wk  staff  7980412 Aug 21 21:57 segmed.test
-rw-rw-r--   1 wk  staff     1139 Aug 21 21:57 tags
-rw-rw-r--   1 wk  staff       70 Aug 21 21:57 unused_to_test.go

I attempt to emulate impact of unused code by making unused_to_test.go with 1 global function not called from anywhere. No luck in reproducing -- this project builds fine with Golang 1.15.

Summary: asset.go itself and probably pkger aren't the main culprit. It appears like a combination of other factors exposes this linker issue.

@wkoszek
Copy link
Author

@wkoszek wkoszek commented Aug 22, 2020

I can repeat go test with additional flags:

script out go test -x -work -o segmed.test -v -cover -c

It fails too, and I can reproduce exact same linker error by running the same command:

[23:31:59][wk@wkoszek_mba:~/r/segmed/dataops/cli]$ grep WORK out | head -3
WORK=/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-build777292036
mkdir -p $WORK/b001/
mkdir -p $WORK/b006/
[23:32:33][wk@wkoszek_mba:~/r/segmed/dataops/cli]$ cd /var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-build777292036
[23:32:39][wk@wkoszek_mba:/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-build777292036]$ export WORK=/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-build777292036
[23:32:48][wk@wkoszek_mba:/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-build777292036]$ /usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64/link -o $WORK/b001/cli.test -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=G8Oc1RIqWiWmA0px7Hxo/v9rR7LoNABb9-9eity6N/QOSmLV_u7J8EEbKnm0Si/G8Oc1RIqWiWmA0px7Hxo -extld=clang /Users/wk/Library/Caches/go-build/c9/c98e9ed85f690a74a9c536418afa91c051f9358c555681e3469597aa2f19b141-d
/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: sectionForAddress(0xA9CE3F) address not in any section file '/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-link-700263833/go.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[23:33:01][wk@wkoszek_mba:/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-build777292036]$
@ianlancetaylor ianlancetaylor changed the title go1.15: ld: sectionForAddress(0xA9D67F) address not in any section file cmd/link: sectionForAddress(0xA9D67F) address not in any section file Aug 22, 2020
@ianlancetaylor ianlancetaylor added this to the Go1.16 milestone Aug 22, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 22, 2020

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 23, 2020

Thanks for the report. Is there any way we can reproduce it ourselves? With the current information it is hard to tell what's wrong.

If you add -ldflags "-w -s" to your go test -c command, does it make any difference? If so, it could be related to debug info for unused functions.

@wkoszek
Copy link
Author

@wkoszek wkoszek commented Aug 23, 2020

@cherrymui Thanks. I attempted to minimize the failing case, but in the time I had, I failed to find anything. But your idea sheds some light, as it works without debugging symbols:

$ go test  -o segmed.test -v -cover -c
# gitlab.com/segmed/dataops/cli.test
/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: sectionForAddress(0xA9CE3F) address not in any section file '/var/folders/gx/4jcx3wqs70195ygr767r2xsr0000gn/T/go-link-232810185/go.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ go test -ldflags "-w -s" -o segmed.test -v -cover -c
$
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 23, 2020

Thanks for the information.

Could you get the intermediate go.o file and get some information about it? One way is to run

mkdir /tmp/xxx
go test -c -ldflags=-tmpdir=/tmp/xxx

and look at /tmp/xxx/go.o (replace /tmp/xxx with a temp directory you like). Maybe run nm -n /tmp/xxx/go.o and otool -r /tmp/xxx/go.o and see if there is any symbol at address 0xA9CE3F, and if there is any relocation targeting that address. (Or, if you don't mind, you can attach the go.o file and I'll look into it.)

@wkoszek
Copy link
Author

@wkoszek wkoszek commented Aug 24, 2020

@cherrymui Thanks. Output:

[15:02:55][wk@wkoszek_mba:/tmp/xxx]$ otool -r go.o > go_o_otool_dash_r_output.txt
[15:03:08][wk@wkoszek_mba:/tmp/xxx]$ nm -n go.o  > go_o_nm_dash_n_out.txt
[15:03:23][wk@wkoszek_mba:/tmp/xxx]$ grep -ri 'a9ce3f' *
go_o_nm_dash_n_out.txt:0000000000a9ce3f t _runtime.etext
[15:03:39][wk@wkoszek_mba:/tmp/xxx]$ la go*
-rwxrwxr-x  1 wk  wheel  48010744 Aug 24 14:44 go.o*
-rw-rw-r--  1 wk  wheel   2505130 Aug 24 15:03 go_o_nm_dash_n_out.txt
-rw-rw-r--  1 wk  wheel  23860264 Aug 24 15:03 go_o_otool_dash_r_output.txt
@cpick
Copy link

@cpick cpick commented Aug 24, 2020

I've seen a nearly identical issue in several of my projects (the error message reports a different address, but otherwise looks to be the same):

$ go build ./cmd/service
# path/to/my/repo/cmd/service
/usr/local/opt/go/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: sectionForAddress(0xA2F51E) address not in any section file '/var/folders/yt/7ql8k1yn67s0qfhcvytsn7g00000gn/T/go-link-948504621/go.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I am also building for macOS natively.

The error goes away if I do any of the following:

  • go build -x -ldflags '-w' ./cmd/service
  • CGO_ENABLED=0 go build ./cmd/service
  • go get -u ./... && go build ./cmd/service

The fact that upgrading all my dependencies seems to fix it is very interesting to me. I haven't had a chance to bisect down to what individual upgrade (if any) fixes the issue.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 24, 2020

Thanks for the information. From what you posted it seems we generated a DWARF relocation targeting runtime.etext (which is not a marker, not a real symbol). I should be able to look into it from here.

@dylan-bourque
Copy link

@dylan-bourque dylan-bourque commented Aug 25, 2020

I'm seeing this same error when running ginkgo -r -cover -coverprofile=coverage.out after upgrading to Go 1.15. Running ginkgo without -cover -coverprofile=coverage.out does not generate the error.

Minimally redacted output

Failed to compile problem-repo:

# github.com/example/problem-repo.test
/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: sectionForAddress(0x7D57FF) address not in any section file '/var/folders/3t/4048hqr5459dw83ksmkqxn7r0000gn/T/go-link-635331615/go.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Ginkgo ran 1 suite in 15.798262466s
Test Suite Failed

My environment

> go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/dbourque/Library/Caches/go-build"
GOENV="/Users/dbourque/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/dbourque/dev/golang/pkg/mod"
GOOS="darwin"
GOPATH="/Users/dbourque/dev/golang"
GOROOT="/usr/local/Cellar/go/1.15/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/dbourque/dev/problem-repo/go.mod"
CGO_CFLAGS=""
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="/usr/local/bin/pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3t/4048hqr5459dw83ksmkqxn7r0000gn/T/go-build878968521=/tmp/go-build -gno-record-gcc-switches -fno-common"

I would share more but this is an internal project.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 26, 2020

Hmmm, there are a few places in DWARF where we reference the end address of text section. But they are the same in Go 1.14 as well. Not sure what changed...

@joesteele
Copy link

@joesteele joesteele commented Sep 10, 2020

I noticed @wkoszek mentioned running into this with a file named asset.go. Incidentally I was running into this on an internal project on a branch where I was adding a package called assets with numerous "asset" related code. I tried renaming the package without luck, before just adding the -w flag to keep moving.

I'm not sure if that's just a red herring and just coincidence (we already had/have at least 2 other files named asset.go in our code base), but I thought I'd mention it in case. None of our other branches have had any issues except that one.

@wkoszek
Copy link
Author

@wkoszek wkoszek commented Sep 17, 2020

@joesteele I don't think that's it. I'm getting it now on a project without a "asset.go", and the bug still exists.

@joesteele
Copy link

@joesteele joesteele commented Sep 18, 2020

Yeah, I figured it was just a coincidence, but when I saw that I was like 🤨 , haha.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.