Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: Unified IR exports table is binary unstable in presence of generics #59571

Closed
nemo-cpt opened this issue Apr 12, 2023 · 8 comments
Closed
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@nemo-cpt
Copy link

nemo-cpt commented Apr 12, 2023

What version of Go are you using (go version)?

$ go version
go version go1.20.2 linux/amd64

Does this issue reproduce with the latest release?

Yes (tested on go 1.20.3)
go1.19.8 works fine even with unified IR enabled

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/nemo-cpt/.cache/go-build"
GOENV="/home/nemo-cpt/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/nemo-cpt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/nemo-cpt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/nemo-cpt/go_root"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/nemo-cpt/go_root/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/nemo-cpt/go_bug/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/var/tmp/go-build1709447254=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Internally we use extensive caching of precompiled go packages, but we started seeing fingerprint mismatch issue after switching to 1.20.

The problem seems to be the following: starting with go 1.20 packages are binary unstable: the fingerprint changes from build to build (of the exactly same code) in presence of generics. The code (as reported by objdump) and the names (as reported by nm) are identical. In our environment we see some shuffling in export table strings related to generic instantiations (the total size of table remains the same). In more generic environment I see wider changes, but I still expect to see none.

Looking at the code of
cmd/compile/internal/noder/unified.go
and
internal/pkgbits/encoder.go

We see that fingerprint is exactly hash of the exports table.

--

The problem is not specific to the project below, but can be reproduced on it:

checkout https://github.com/urfave/cli, than do the following:

# Build the same package twice
$GOROOT/bin/go build -o output_1.a -a
$GOROOT/bin/go build -o output_2.a -a
# produce hexdump for comparson
hexdump -C  output_1.a > ../1.hex
hexdump -C  output_2.a > ../2.hex
# Compare
diff ../1.hex ../2.hex

What did you expect to see?

Binary exact packages (as with 1.19). At the very least difference that does not affect package fingerprint.

What did you see instead?

...
10722,10723c10722,10723
< 0002a460  01 01 00 01 00 54 0b 05  06 00 01 00 55 01 e6 1d  |.....T......U...|
< 0002a470  19 99 fc 0f 35 6d 0a 24  24 0a 5f 67 6f 5f 2e 6f  |....5m.$$._go_.o|
---
> 0002a460  01 01 00 01 00 54 0b 05  06 00 01 00 55 01 05 57  |.....T......U..W|
> 0002a470  74 10 9a 07 c2 57 0a 24  24 0a 5f 67 6f 5f 2e 6f  |t....W.$$._go_.o|


10735,10737c10735,10737
< 0002a530  39 50 6d 2f 4c 57 5a 43  56 73 39 46 57 73 65 78  |9Pm/LWZCVs9FWsex|
< 0002a540  71 79 4b 30 45 39 73 6a  22 0a 0a 0a 21 0a 00 67  |qyK0E9sj"...!..g|
< 0002a550  6f 31 32 30 6c 64 e6 1d  19 99 fc 0f 35 6d 00 00  |o120ld......5m..|
---
> 0002a530  39 50 6d 2f 74 4c 74 4b  72 55 48 57 4c 6c 5f 30  |9Pm/tLtKrUHWLl_0|
> 0002a540  78 77 76 74 4b 61 58 5f  22 0a 0a 0a 21 0a 00 67  |xwvtKaX_"...!..g|
> 0002a550  6f 31 32 30 6c 64 05 57  74 10 9a 07 c2 57 00 00  |o120ld.Wt....W..|

According to source code

  • Last 8 bytes before \n$$\n of export table are fingerprint
  • 8bytes after go120ld are the package fingerprint.
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 12, 2023
@randall77
Copy link
Contributor

@mdempsky

@mdempsky mdempsky self-assigned this Apr 12, 2023
@mdempsky mdempsky added the NeedsFix The path to resolution is known, but the work has not been done. label Apr 12, 2023
@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

Repro case:

package cli

type IntSliceFlag struct{}

type StringSliceFlag struct{}

type IntSlice struct{}

type StringSlice struct {
}

type (
	SliceFlag[T SliceFlagTarget[E], S ~[]E, E any] struct{}
	SliceFlagTarget[E any]                         interface{}
	MultiStringFlag                                = SliceFlag[*StringSliceFlag, []string, string]
	MultiIntFlag                                   = SliceFlag[*IntSliceFlag, []int, int]
)

func (x *SliceFlag[T, S, E]) String() string { return "zzz" }
% for (i in `{seq 10}) go build -a -o x.a && openssl sha256 x.a 
SHA256(x.a)= d310fb5a5b329714bcc88ddf9056fa20524a0413a37cdb30122bc974ae415651
SHA256(x.a)= 119fb8ceed9ca6c9c621bf6f4348b083a0de2eaf98951885bd2e390d6573c8ce
SHA256(x.a)= d310fb5a5b329714bcc88ddf9056fa20524a0413a37cdb30122bc974ae415651
SHA256(x.a)= d310fb5a5b329714bcc88ddf9056fa20524a0413a37cdb30122bc974ae415651
SHA256(x.a)= 119fb8ceed9ca6c9c621bf6f4348b083a0de2eaf98951885bd2e390d6573c8ce
SHA256(x.a)= d310fb5a5b329714bcc88ddf9056fa20524a0413a37cdb30122bc974ae415651
SHA256(x.a)= d310fb5a5b329714bcc88ddf9056fa20524a0413a37cdb30122bc974ae415651
SHA256(x.a)= d310fb5a5b329714bcc88ddf9056fa20524a0413a37cdb30122bc974ae415651
SHA256(x.a)= d310fb5a5b329714bcc88ddf9056fa20524a0413a37cdb30122bc974ae415651
SHA256(x.a)= d310fb5a5b329714bcc88ddf9056fa20524a0413a37cdb30122bc974ae415651

@mdempsky
Copy link
Contributor

Further minimized:

package cli

type (
	SliceFlag[T any] struct{}
	MultiStringFlag  = SliceFlag[string]
	MultiIntFlag     = SliceFlag[int]
)

func (x *SliceFlag[T]) String() string { return "zzz" }

@rsc
Copy link
Contributor

rsc commented Apr 12, 2023

Assuming the fix is a trivial change like a strategically placed call to sort.Sort, we should backport the fix to Go 1.20.

@mdempsky
Copy link
Contributor

@rsc Agreed. I've narrowed down the cause; just figuring out the best way to fix it.

@gopherbot Please backport to Go 1.20.

@gopherbot
Copy link
Contributor

Backport issue(s) opened: #59585 (for 1.20).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/484155 mentions this issue: cmd/compile: fix reproducible build of aliased generic types

@dmitshur dmitshur added this to the Go1.21 milestone Apr 24, 2023
@dolmen
Copy link
Contributor

dolmen commented May 10, 2023

Well done @mdempsky @rsc .

@golang golang locked and limited conversation to collaborators May 9, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

7 participants