cmd/compile: GOEXPERIMENT=unified on macOS fails fixedbugs/issue27836.go

[dr2chase]

What version of Go are you using (go version)?

$ go version
go version devel go1.19-de8101d21b Mon Jul 18 18:04:23 2022 +0000 X:unified darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/drchase/Library/Caches/go-build"
GOENV="/Users/drchase/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT="unified"
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/drchase/work/gocode/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/drchase/work/gocode"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/drchase/work/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/drchase/work/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="devel go1.19-de8101d21b Mon Jul 18 18:04:23 2022 +0000 X:unified"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v6/9yfw749x623dr55rpm6ntnsc0095tn/T/go-build1756414274=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

GOEXPERIMENT=unified ./make.bash
cd ../test
go run run.go -- fixedbugs/issue27836.go

What did you expect to see?

Success

What did you see instead?

<unknown line number>: internal compiler error: have package "test/Äfoo" (0xc00044fd10), want package "test/Äfoo" (0xc00044fb80)

[dr2chase]

I'm going to start bisecting. I tried this on a hunch that something had changed in tip (since the dev.unified branch was working, till the latest merge).

[bcmills]

The compiledir action in run.go has a goDirPackages helper function to associate the file into packages, and that function uses ioutil.ReadFile to parse the actual package line from the source code. So the output of ReadDir shouldn't actually matter, because we're using the source code — not the filename.

I think the bug is here:
https://cs.opensource.google/go/go/+/master:test/run.go;l=272;drc=1243ec9c177007879958443262fe4d25099c5ede
We're using names[0] instead of pkgname to compute the import path to pass to the compiler. (names[0] is the filename from ReadDir, not the actual package name parsed from the file.)

That suggests https://go.dev/cl/395258 as the culprit (attn @mdempsky). Does March 24 sound like a plausible timeframe?

[dr2chase]

It was working on May 2, at the boringcrypto merge 0668e3c .
That still might be part of the problem, but I want to see what bisecting finds, now that I have good and bad points.

[dr2chase]

So the other half of the problem, no surprise, was cmd/compile: set LocalPkg.Path to -p flag.

[mdempsky]

We're using names[0] instead of pkgname to compute the import path to pass to the compiler.

But package path and package name are separate names. The test is structured so you can vary them independently, and we allow them to vary separately in normal usage too (eg, packages within GOPATH or modules).

The problem I think is issue27836 relies on the file system preserving Unicode file names exact byte encoding, but macOS canonicalizes file names in some cases. I would expect the same problem could arise if a Go module had a similarly named directory: it would be compiled as one path on macOS due to canonicalization, and a different one on other OSes.

[gopherbot]

Change https://go.dev/cl/418294 mentions this issue: test: change Unicode file/package name to use characters not translated by macOS.

[bcmills]

I would expect the same problem could arise if a Go module had a similarly named directory: it would be compiled as one path on macOS due to canonicalization, and a different one on other OSes.

That shouldn't be the case (modulo bugs, of course), but see also #38342 and #38571. 🙃

[gopherbot]

Change https://go.dev/cl/418334 mentions this issue: [dev.unified] test: change Unicode file/package name to use characters not translated by macOS.