Bootstrap fails on AlpineLinux edge

[folliehiyuki]

I'm trying to install please on my local machine, running AlpineLinux edge.

The command I use, based on the CI config:

go run src/please.go -v debug --profile alpine --exclude no-musl --log_file plz-out/log/boostrap_build.log build //src:please

My environment variable from go env:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/follie/.cache/go-build"
GOENV="/home/follie/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/follie/.local/share/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/follie/.local/share/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/follie/Code/please/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1508056060=/tmp/go-build -gno-record-gcc-switches"

The build is currently failing with these logs:

...
21:15:36.923   DEBUG: Building target ///third_party/go/github.com_alessio_shellescape//:shellescape
ENVIRONMENT:
LANG=en_GB.UTF-8
ARCH=amd64
OS=linux
XARCH=x86_64
XOS=linux
PATH=/tmp/go-build1230943462/b001/exe:/usr/local/bin:/usr/bin:/bin
PKG=
PKG_DIR=pkg/linux_amd64/github.com/alessio/shellescape
NAME=shellescape
BUILD_CONFIG=opt
CONFIG=opt
TMP_DIR=/home/follie/Code/please/plz-out/tmp/third_party/go/github.com_alessio_shellescape/shellescape._build
TMPDIR=/home/follie/Code/please/plz-out/tmp/third_party/go/github.com_alessio_shellescape/shellescape._build
SRCS=pkg/linux_amd64/github.com/alessio/shellescape/shellescape.go
OUTS=shellescape.a
HOME=/home/follie/Code/please/plz-out/tmp/third_party/go/github.com_alessio_shellescape/shellescape._build
PYTHONHASHSEED=42
OUT=/home/follie/Code/please/plz-out/tmp/third_party/go/github.com_alessio_shellescape/shellescape._build/shellescape.a
SRC=pkg/linux_amd64/github.com/alessio/shellescape/shellescape.go
SRCS_GO=pkg/linux_amd64/github.com/alessio/shellescape/shellescape.go
TOOLS=/usr/bin/go /home/follie/Code/please/plz-out/bin/go/tools/please_go/please_go.bootstrap
TOOLS_PLEASE_GO=/home/follie/Code/please/plz-out/bin/go/tools/please_go/please_go.bootstrap
TOOLS_GO=/usr/bin/go
RULE_HASH=4ERdUtFSRN8aZoNw6a_9TEhuPpA
export GOPATH=$TMP_DIR && export GOROOT=$("$TOOLS_GO" env GOROOT) && find "$GOROOT"/pkg/linux_amd64 -name "*.a" | sed -e s=^"$GOROOT"/pkg/linux_amd64/== | sed -e s="\.a$"== | xargs -I{} echo "packagefile {}="$GOROOT"/pkg/linux_amd64/{}.a" | sort -u > goroot.importconfig && find . -name "*.importconfig" | xargs -I{} cat {} | sort -u >> importconfig; export SRCS_GO="$("${TOOLS_PLEASE_GO}" filter ${SRCS_GO})"; "$TOOLS_GO" tool compile -importcfg importconfig -trimpath "$TMP_DIR" -complete  -pack  -o "$OUT" -p github.com/alessio/shellescape $SRCS_GO
21:15:36.923    INFO: [489/734 tasks] //third_party/go:_google.golang.org_genproto#repo: Unchanged [0.0s]
21:15:36.930    INFO: ///third_party/go/github.com_alessio_shellescape//:shellescape: Compiling...
21:15:36.930    INFO: [490/734 tasks] //src/assets:assets: Build failed: Error building target //src/assets:assets: exit status 1
find: /usr/lib/go/pkg/linux_amd64: No such file or directory
compile: -importcfg: open importconfig: no such file or directory
 [0.0s]
21:15:36.930   ERROR: //src/assets:assets failed:�[0m
Error building target //src/assets:assets: exit status 1
find: /usr/lib/go/pkg/linux_amd64: No such file or directory
compile: -importcfg: open importconfig: no such file or directory
21:15:36.930    INFO: [490/734 tasks] //rules/bazel:bazel: Build failed: Error building target //rules/bazel:bazel: exit status 1
find: /usr/lib/go/pkg/linux_amd64: No such file or directory
compile: -importcfg: open importconfig: no such file or directory
 [0.0s]
21:15:36.930   ERROR: //rules/bazel:bazel failed:�[0m
Error building target //rules/bazel:bazel: exit status 1
find: /usr/lib/go/pkg/linux_amd64: No such file or directory
compile: -importcfg: open importconfig: no such file or directory
21:15:36.930    INFO: [490/734 tasks] //rules:rules: Build failed: Error building target //rules:rules: exit status 1
find: /usr/lib/go/pkg/linux_amd64: No such file or directory
compile: -importcfg: open importconfig: no such file or directory
 [0.0s]
21:15:36.930   ERROR: //rules:rules failed:�[0m
Error building target //rules:rules: exit status 1
find: /usr/lib/go/pkg/linux_amd64: No such file or directory
compile: -importcfg: open importconfig: no such file or directory
21:15:36.922   DEBUG: Read config from plz-out/subrepos/third_party/go/google.golang.org_genproto/.plzconfig
21:15:36.993    INFO: Shutting down cache workers...
21:15:36.994    INFO: [634/1213 tasks] ///third_party/go/github.com_alessio_shellescape//:shellescape: Build failed: Error building target ///third_party/go/github.com_alessio_shellescape//:shellescape: exit status 1
find: /usr/lib/go/pkg/linux_amd64: No such file or directory
compile: -importcfg: open importconfig: no such file or directory
 [0.1s]
21:15:36.994   ERROR: ///third_party/go/github.com_alessio_shellescape//:shellescape failed:�[0m
Error building target ///third_party/go/github.com_alessio_shellescape//:shellescape: exit status 1
find: /usr/lib/go/pkg/linux_amd64: No such file or directory
compile: -importcfg: open importconfig: no such file or directory
21:15:36.994   DEBUG: Shut down all cache workers
21:15:36.997   DEBUG: Metric recorded: go_memstats_alloc_bytes_total: 145325120
21:15:36.998   DEBUG: Metric recorded: go_memstats_frees_total: 858514
21:15:36.998   DEBUG: Metric recorded: go_memstats_lookups_total: 0
21:15:36.998   DEBUG: Metric recorded: go_memstats_mallocs_total: 1882585
21:15:36.998   DEBUG: Metric recorded: plz_remote_dirs_downloaded_total: 0
21:15:36.998   DEBUG: Metric recorded: plz_remote_dirs_retrieved_total: 0
21:15:36.998   DEBUG: Metric recorded: plz_remote_dirs_stored_total: 0
21:15:36.998   DEBUG: Metric recorded: plz_remote_tree_digest_download_errors_total: 0
21:15:36.998   DEBUG: Metric recorded: process_cpu_seconds_total: 1

I don't know where it generated "$GOROOT"/pkg/linux_amd64 from but that path doesn't exist on my machine.

[mibzman]

I'm also having this issue. Following the codelab, when I try to build a hello world project I get:

Build stopped after 120ms. 1 target failed:
    //src:_main#lib
Error building target //src:_main#lib: exit status 1
find: '/usr/local/go/pkg/linux_amd64': No such file or directory
compile: -importcfg: open importconfig: no such file or directory

[Tatskaari]

Hey all! Thanks for the bug report. The alpine profile uses Go from the host machine and in 1.20 they removed the compiled version of the SDK. I've updated the codelabs to reflect this:

https://please.build/codelabs/go_intro/#3

Essentially to compile on alpine edge, you need to run GODEBUG="installgoroot=all" go install std, which will create the missing files.

[folliehiyuki]

From go 1.20 changelog:

Instead, packages in the standard library are built as needed and
cached in the build cache, just like packages outside GOROOT. This
change reduces the size of the Go distribution and also avoids C
toolchain skew for packages that use cgo. 

I think this is a good decision from Go that we should follow. Is there any reason we can't make please to be built with the default go toolchain (without go install std hack)?

[folliehiyuki]

I understand that having stdlib precompiled is a good thing for a build system like please, to avoid breaking build cache. Even so, I'd like to be able to use both my system's Go toolchain and the Go toolchain from Please as is without any hacks.

Maybe we can do something like what Bazel does here: bazelbuild/rules_go#3385 (pre-compile those files on-demand before the actual build action).

[Tatskaari]

Oh, we actually have this stdlib build rule that does that:
https://github.com/please-build/go-rules/blob/master/build_defs/go.build_defs#L131

I believe if you add that to your repo, set the GoTool opttion to just go. but set STDLib config option to the above rule, Please should compile your toolchain for you, including any fancy build mode or architecture you need.

I should probably suggest that in the codelab instead of the hack. That was an oversite.

[folliehiyuki]

Oh it's true, I can get please to build with this diff:

diff --git a/.plzconfig.alpine b/.plzconfig.alpine
index a3f43db..1072a9a 100644
--- a/.plzconfig.alpine
+++ b/.plzconfig.alpine
@@ -1,6 +1,7 @@
 [Plugin "go"]
 DefaultStatic = true
 GoTool = go
+Stdlib = //third_party/go:std
 
 [buildconfig]
 static-sandbox = true
diff --git a/third_party/go/BUILD b/third_party/go/BUILD
index ad4f17e..e3eb754 100644
--- a/third_party/go/BUILD
+++ b/third_party/go/BUILD
@@ -14,6 +14,8 @@ go_toolchain(
     version = "1.19",
 )
 
+go_stdlib(name = "std")
+
 go_mod_download(
     name = "build_tools_dl",
     module = "github.com/peterebden/buildtools",

Would it be ok if I open a PR to include this here? Then, the hack can be removed from the codelab docs.

[folliehiyuki]

Oh never mind there is an opened PR including this diff: https://github.com/thought-machine/please/pull/2803/files. I'll wait for the next release then.