Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

race: not working with Alpine based image #14481

Open
dlsniper opened this issue Feb 23, 2016 · 25 comments

Comments

@dlsniper
Copy link
Contributor

commented Feb 23, 2016

Hi,

With the following docker image (save it as demo.docker)

FROM golang:1.6.0-alpine
MAINTAINER "florinpatan@gmail.com"

RUN apk add --update alpine-sdk \
    && rm -rf /var/cache/apk/*

run

docker build -f demo.docker -t race/demo .

Then you can finally run the command:

PROJECT_DIR="${PWD}" #assume we are in $GOPATH/src/github.com/dlsniper/demo on the computer
CONTAINER_PROJECT_DIR="/go/src/github.com/dlsniper/demo"
CONTAINER_PROJECT_GOPATH="${CONTAINER_PROJECT_DIR}/vendor:/go"

docker run --rm \
        --net="host" \
        -v ${PROJECT_DIR}:${CONTAINER_PROJECT_DIR} \
        -e CI=true \
        -e GODEBUG=netdns=go \
        -e CGO_ENABLED=1 \
        -e GOPATH=${CONTAINER_PROJECT_GOPATH} \
        -w "${CONTAINER_PROJECT_DIR}" \
        race/demo \
        go test -v -race ./...

This will fail with:

# runtime/race
race_linux_amd64.syso: In function `__sanitizer::InternalAlloc(unsigned long, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<0ul, 140737488355328ull, 0ul, __sanitizer::SizeClassMap<17ul, 64ul, 14ul>, 20ul, __sanitizer::TwoLevelByteMap<32768ull, 4096ull, __sanitizer::NoOpMapUnmapCallback>, __sanitizer::NoOpMapUnmapCallback> >*)':
gotsan.cc:(.text+0x1681): undefined reference to `__libc_malloc'
race_linux_amd64.syso: In function `__sanitizer::ReExec()':
gotsan.cc:(.text+0xd937): undefined reference to `__libc_stack_end'
race_linux_amd64.syso: In function `__sanitizer::InternalFree(void*, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<0ul, 140737488355328ull, 0ul, __sanitizer::SizeClassMap<17ul, 64ul, 14ul>, 20ul, __sanitizer::TwoLevelByteMap<32768ull, 4096ull, __sanitizer::NoOpMapUnmapCallback>, __sanitizer::NoOpMapUnmapCallback> >*)':
gotsan.cc:(.text+0x5ec8): undefined reference to `__libc_free'
collect2: error: ld returned 1 exit status

If you then disable CGO and run again:

PROJECT_DIR="${PWD}" #assume we are in $GOPATH/src/github.com/dlsniper/demo on the computer
CONTAINER_PROJECT_DIR="/go/src/github.com/dlsniper/demo"
CONTAINER_PROJECT_GOPATH="${CONTAINER_PROJECT_DIR}/vendor:/go"

docker run --rm \
        --net="host" \
        -v ${PROJECT_DIR}:${CONTAINER_PROJECT_DIR} \
        -e CI=true \
        -e GODEBUG=netdns=go \
        -e CGO_ENABLED=0 \
        -e GOPATH=${CONTAINER_PROJECT_GOPATH} \
        -w "${CONTAINER_PROJECT_DIR}" \
        race/demo \
        go test -v -race ./...

It will result in the following output:

go test: -race requires cgo; enable cgo by setting CGO_ENABLED=1

Previously, in go 1.5.3, when running with CGO disabled, this used to fail with:

# testmain
runtime/race(.text): __libc_malloc: not defined
runtime/race(.text): getuid: not defined
runtime/race(.text): pthread_self: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): sleep: not defined
runtime/race(.text): usleep: not defined
runtime/race(.text): abort: not defined
runtime/race(.text): isatty: not defined
runtime/race(.text): __libc_free: not defined
runtime/race(.text): getrlimit: not defined
runtime/race(.text): pipe: not defined
runtime/race(.text): __libc_stack_end: not defined
runtime/race(.text): getrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): exit: not defined
runtime/race(.text.unlikely): __errno_location: not defined
runtime/race(.text): undefined: __libc_malloc
/usr/local/go/pkg/tool/linux_amd64/link: too many errors

To test this just change the base image for the container.

Please let me know if there are any additional details I can share.

Thank you.

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Feb 23, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 23, 2016

CC @dvyukov

As far as I know this is because race.syso assumes a glibc based system. I don't know that there is a simple fix here.

@dvyukov

This comment has been minimized.

Copy link
Member

commented Feb 24, 2016

I can't think of any simple fix. A complex fix would be to remove all dependencies on libc from race runtime (there is an open issue for that).
alpinelinux wiki suggests that there are some ways to run glibc-based programs on alpine:
http://wiki.alpinelinux.org/wiki/Running_glibc_programs
Don't know whether it will help for race detector or not.

@mbana

This comment has been minimized.

Copy link

commented Feb 5, 2017

hi, all,

just wondering. does anyone have a complete working example of a golang program working with -race ideally not that much different from the official Alpine-based build.

out of interest, are the flags not tested? i would imagine that this sort of issue would be picked up at compile time, unless the Alpine-based is configured to to include it... no idea, just guessing.

anywho, working sample would be much appreciated.

@neelance

This comment has been minimized.

Copy link
Member

commented Feb 23, 2017

I have managed to build a race_linux_amd64.syso that works on Alpine. Here are the necessary changes: neelance/compiler-rt@32aa655 I haven't verified that it still works on other Linux platforms.

To build on Alpine:

  1. Clone https://github.com/neelance/compiler-rt/
  2. Go into /compiler-rt/lib/tsan/go/
  3. ./buildgo.sh (The test step crashes, but the library works anyways. My guess is that the test does not use a proper Go environment.)
  4. Copy race_linux_amd64.syso to $GOPATH/src/runtime/race/

I'm not sure how to continue from here. Any suggestions on getting that upstream?

@dvyukov

This comment has been minimized.

Copy link
Member

commented Feb 24, 2017

Here are instructions on how to contribute to sanitizers:
https://github.com/google/sanitizers/wiki/AddressSanitizerHowToContribute

I skimmed through your patch and I think we can upstream something like this. But we need some testing story, otherwise it will break in future.

One good test might be to whitelist all libc symbol dependencies in buildgo.sh. This way we can ensure that we won't silently add a new dependency (ideally in future this list becomes empty).

@neelance

This comment has been minimized.

Copy link
Member

commented Feb 24, 2017

@dvyukov I see that you did most of the updates of the syso files. I have some more questions:

  1. How can we test the patch properly on the Go end across all platforms so we don't upstream something stupid?
  2. Can we somehow add Alpine to the platforms being tested?
  3. We need to fix that crash on the test phase of buildgo.sh. I've investigated it a bit, seems like mmap is failing for some shadow memory. My wild guess right now is that it expects a Go memory layout, but the test is a C program. That was as far as I could get without spending a massive amount of time to understand how that thread sanitizer works.
@dvyukov

This comment has been minimized.

Copy link
Member

commented Feb 24, 2017

How can we test the patch properly on the Go end across all platforms so we don't upstream something stupid?

I now use golang.org/x/build/cmd/racebuild which builds and tests race runtime for all platforms. But it requires gomote access (I think you need to be a committer for Go repo). If you don't have that access, I will do testing. But test at least on Alpine and on glibc-based Linux.

Can we somehow add Alpine to the platforms being tested?

You need to either run your own Go builder and connect it to dashboard, or we need to setup Alpine linux builder on GCE. @bradfitz, do we want to setup/maintain it?

seems like mmap is failing for some shadow memory. My wild guess right now is that it expects a Go memory layout

It's possible. Does removing -fPIC -fpie from buildgo.sh help? We still need to build the runtime with -pie, but the test itself does not need to be pie.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Feb 24, 2017

#17891 tracks adding an Alpine builder. @jessfraz has a (stalled :)) CL in progress.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Apr 25, 2017

We now have Alpine builders, so in theory golang.org/x/build/cmd/racebuild could build an Alpine race binary now.

For now I'm just going to be skipping race tests on Alpine so we don't regress on other things.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 25, 2017

CL https://golang.org/cl/41678 mentions this issue.

gopherbot pushed a commit that referenced this issue Apr 25, 2017
cmd/go, cmd/dist: temporarily disable race and PIE internal link test…
…s on Alpine

In an effort to at least understand the complete set of things not
working on Alpine Linux, I've been trying to get the build passing
again, even with tests disabled.

The race detector is broken on Alpine. That is #14481 (and #9918).
So disable those tests for now.

Also, internal linking with PIE doesn't work on Alpine yet.
That is #18243. So disable that test for now.

With this CL, all.bash almost passes. There's some cgo test failing
still, but there's no bug yet, so that can be a separate CL.

Change-Id: I3ffbb0e787ed54cb82f298b6bd5bf3ccfbc82622
Reviewed-on: https://go-review.googlesource.com/41678
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
benesch added a commit to benesch/cockroach that referenced this issue Apr 26, 2017
build: don't (fail to) build musl race binaries
The race detector doesn't work without glibc due to golang/go#14481.

Fixes cockroachdb#15378.
AkihiroSuda pushed a commit to AkihiroSuda/linuxkit that referenced this issue Jun 6, 2017
Akihiro Suda
this won't work on alpine
golang/go#14481

Signed-off-by: Akihiro Suda <suda@suda-mbp.osrg.net>
camh-anz pushed a commit to camh-anz/go-course that referenced this issue Feb 2, 2019
cloudbuild: Enable race detector on tests
Go has a race detector enabled by running `go` with `-race`. This was
disabled for Google cloudbuild because it didn't work because the
cloud-builder used was alpine-based, and the race detector does not work
on alpine (golang/go#14481). The race detector
needs glibc but alpine uses musl.

Switch to a debian-based cloud-builder image instead. Debian has glibc
so the race detector works.

Issues: anz-bank#14
camh-anz added a commit to camh-anz/go-course that referenced this issue Feb 2, 2019
cloudbuild: Enable race detector on tests
Go has a race detector enabled by running `go` with `-race`. This was
disabled for Google cloudbuild because it didn't work because the
cloud-builder used was alpine-based, and the race detector does not work
on alpine (golang/go#14481). The race detector
needs glibc but alpine uses musl.

Switch to a debian-based cloud-builder image instead. Debian has glibc
so the race detector works.

Issues: anz-bank#14
camh-anz added a commit to camh-anz/go-course that referenced this issue Feb 2, 2019
cloudbuild: Enable race detector on tests
Go has a race detector enabled by running `go` with `-race`. This was
disabled for Google cloudbuild because it didn't work because the
cloud-builder used was alpine-based, and the race detector does not work
on alpine (golang/go#14481). The race detector
needs glibc but alpine uses musl.

Switch to a debian-based cloud-builder image instead. Debian has glibc
so the race detector works.

Issues: anz-bank#14
imsky added a commit to imsky/github-fresh that referenced this issue Apr 25, 2019
imsky added a commit to imsky/github-fresh that referenced this issue Apr 25, 2019
amuttsch added a commit to amuttsch/go-swagger that referenced this issue May 17, 2019
Use golang:stretch as base image for CI
Alpine doesn't support go test -race due to musl c library:
golang/go#14481

Signed-off-by: Andreas Muttscheller <andreas.muttscheller@codecentric.de>
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
nhooyr added a commit to nhooyr/websocket that referenced this issue Aug 21, 2019
Switch to arch linux for CI
Race detector doesn't work on alpine.

See golang/go#14481
@dnwe

This comment has been minimized.

Copy link

commented Sep 16, 2019

In case it helps anyone else, I rebased @neelance's compiler-rt changes (#14481 (comment)) on top of compiler-rt@fe2c72c5 (as used by Go 1.13) and have been able to run -race using:

FROM golang:1.13-alpine3.10 as golang

# Make go test -race work on alpine by building (patched) sanitizer manually
# as it is not built by default
# Ref: https://github.com/golang/go/issues/14481#issuecomment-281972886
# SHA: https://github.com/golang/go/blob/go1.13/src/runtime/race/README
COPY 0001-hack-to-make-Go-s-race-flag-work-on-Alpine.patch /race.patch
RUN cd / \
    && apk add --no-cache --virtual .build-deps g++ git \
    && mkdir -p compiler-rt \
    && git clone https://llvm.org/git/compiler-rt.git \
    && cd compiler-rt \
    && git reset --hard fe2c72c59aa7f4afa45e3f65a5d16a374b6cce26 \
    && patch -p1 -i /race.patch \
    && cd lib/tsan/go \
    && ./buildgo.sh 2>/dev/null \
    && cp -v race_linux_amd64.syso /usr/local/go/src/runtime/race/ \
    && rm -rf /compiler-rt /race.patch \
    && apk del .build-deps

0001-hack-to-make-Go-s-race-flag-work-on-Alpine.patch.gz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.