race: not working with Alpine based image #14481

Open
dlsniper opened this Issue Feb 23, 2016 · 12 comments

Comments

Projects
None yet
8 participants
@dlsniper
Contributor

dlsniper commented Feb 23, 2016

Hi,

With the following docker image (save it as demo.docker)

FROM golang:1.6.0-alpine
MAINTAINER "florinpatan@gmail.com"

RUN apk add --update alpine-sdk \
    && rm -rf /var/cache/apk/*

run

docker build -f demo.docker -t race/demo .

Then you can finally run the command:

PROJECT_DIR="${PWD}" #assume we are in $GOPATH/src/github.com/dlsniper/demo on the computer
CONTAINER_PROJECT_DIR="/go/src/github.com/dlsniper/demo"
CONTAINER_PROJECT_GOPATH="${CONTAINER_PROJECT_DIR}/vendor:/go"

docker run --rm \
        --net="host" \
        -v ${PROJECT_DIR}:${CONTAINER_PROJECT_DIR} \
        -e CI=true \
        -e GODEBUG=netdns=go \
        -e CGO_ENABLED=1 \
        -e GOPATH=${CONTAINER_PROJECT_GOPATH} \
        -w "${CONTAINER_PROJECT_DIR}" \
        race/demo \
        go test -v -race ./...

This will fail with:

# runtime/race
race_linux_amd64.syso: In function `__sanitizer::InternalAlloc(unsigned long, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<0ul, 140737488355328ull, 0ul, __sanitizer::SizeClassMap<17ul, 64ul, 14ul>, 20ul, __sanitizer::TwoLevelByteMap<32768ull, 4096ull, __sanitizer::NoOpMapUnmapCallback>, __sanitizer::NoOpMapUnmapCallback> >*)':
gotsan.cc:(.text+0x1681): undefined reference to `__libc_malloc'
race_linux_amd64.syso: In function `__sanitizer::ReExec()':
gotsan.cc:(.text+0xd937): undefined reference to `__libc_stack_end'
race_linux_amd64.syso: In function `__sanitizer::InternalFree(void*, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<0ul, 140737488355328ull, 0ul, __sanitizer::SizeClassMap<17ul, 64ul, 14ul>, 20ul, __sanitizer::TwoLevelByteMap<32768ull, 4096ull, __sanitizer::NoOpMapUnmapCallback>, __sanitizer::NoOpMapUnmapCallback> >*)':
gotsan.cc:(.text+0x5ec8): undefined reference to `__libc_free'
collect2: error: ld returned 1 exit status

If you then disable CGO and run again:

PROJECT_DIR="${PWD}" #assume we are in $GOPATH/src/github.com/dlsniper/demo on the computer
CONTAINER_PROJECT_DIR="/go/src/github.com/dlsniper/demo"
CONTAINER_PROJECT_GOPATH="${CONTAINER_PROJECT_DIR}/vendor:/go"

docker run --rm \
        --net="host" \
        -v ${PROJECT_DIR}:${CONTAINER_PROJECT_DIR} \
        -e CI=true \
        -e GODEBUG=netdns=go \
        -e CGO_ENABLED=0 \
        -e GOPATH=${CONTAINER_PROJECT_GOPATH} \
        -w "${CONTAINER_PROJECT_DIR}" \
        race/demo \
        go test -v -race ./...

It will result in the following output:

go test: -race requires cgo; enable cgo by setting CGO_ENABLED=1

Previously, in go 1.5.3, when running with CGO disabled, this used to fail with:

# testmain
runtime/race(.text): __libc_malloc: not defined
runtime/race(.text): getuid: not defined
runtime/race(.text): pthread_self: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): madvise: not defined
runtime/race(.text): sleep: not defined
runtime/race(.text): usleep: not defined
runtime/race(.text): abort: not defined
runtime/race(.text): isatty: not defined
runtime/race(.text): __libc_free: not defined
runtime/race(.text): getrlimit: not defined
runtime/race(.text): pipe: not defined
runtime/race(.text): __libc_stack_end: not defined
runtime/race(.text): getrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): setrlimit: not defined
runtime/race(.text): exit: not defined
runtime/race(.text.unlikely): __errno_location: not defined
runtime/race(.text): undefined: __libc_malloc
/usr/local/go/pkg/tool/linux_amd64/link: too many errors

To test this just change the base image for the container.

Please let me know if there are any additional details I can share.

Thank you.

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Feb 23, 2016

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Feb 23, 2016

Contributor

CC @dvyukov

As far as I know this is because race.syso assumes a glibc based system. I don't know that there is a simple fix here.

Contributor

ianlancetaylor commented Feb 23, 2016

CC @dvyukov

As far as I know this is because race.syso assumes a glibc based system. I don't know that there is a simple fix here.

@dvyukov

This comment has been minimized.

Show comment
Hide comment
@dvyukov

dvyukov Feb 24, 2016

Member

I can't think of any simple fix. A complex fix would be to remove all dependencies on libc from race runtime (there is an open issue for that).
alpinelinux wiki suggests that there are some ways to run glibc-based programs on alpine:
http://wiki.alpinelinux.org/wiki/Running_glibc_programs
Don't know whether it will help for race detector or not.

Member

dvyukov commented Feb 24, 2016

I can't think of any simple fix. A complex fix would be to remove all dependencies on libc from race runtime (there is an open issue for that).
alpinelinux wiki suggests that there are some ways to run glibc-based programs on alpine:
http://wiki.alpinelinux.org/wiki/Running_glibc_programs
Don't know whether it will help for race detector or not.

@cyli cyli referenced this issue in theupdateframework/notary Nov 14, 2016

Merged

Do not build the osx cross packages when running CI tests #1032

@kamilsk kamilsk referenced this issue in kamilsk/shared Dec 5, 2016

Closed

fix -race flag in alpine-gcc #13

@mbana

This comment has been minimized.

Show comment
Hide comment
@mbana

mbana Feb 5, 2017

hi, all,

just wondering. does anyone have a complete working example of a golang program working with -race ideally not that much different from the official Alpine-based build.

out of interest, are the flags not tested? i would imagine that this sort of issue would be picked up at compile time, unless the Alpine-based is configured to to include it... no idea, just guessing.

anywho, working sample would be much appreciated.

mbana commented Feb 5, 2017

hi, all,

just wondering. does anyone have a complete working example of a golang program working with -race ideally not that much different from the official Alpine-based build.

out of interest, are the flags not tested? i would imagine that this sort of issue would be picked up at compile time, unless the Alpine-based is configured to to include it... no idea, just guessing.

anywho, working sample would be much appreciated.

@andyxning andyxning referenced this issue in kubernetes/kube-state-metrics Feb 8, 2017

Merged

fix make test units with cgo #81

@neelance

This comment has been minimized.

Show comment
Hide comment
@neelance

neelance Feb 23, 2017

Member

I have managed to build a race_linux_amd64.syso that works on Alpine. Here are the necessary changes: neelance/compiler-rt@32aa655 I haven't verified that it still works on other Linux platforms.

To build on Alpine:

  1. Clone https://github.com/neelance/compiler-rt/
  2. Go into /compiler-rt/lib/tsan/go/
  3. ./buildgo.sh (The test step crashes, but the library works anyways. My guess is that the test does not use a proper Go environment.)
  4. Copy race_linux_amd64.syso to $GOPATH/src/runtime/race/

I'm not sure how to continue from here. Any suggestions on getting that upstream?

Member

neelance commented Feb 23, 2017

I have managed to build a race_linux_amd64.syso that works on Alpine. Here are the necessary changes: neelance/compiler-rt@32aa655 I haven't verified that it still works on other Linux platforms.

To build on Alpine:

  1. Clone https://github.com/neelance/compiler-rt/
  2. Go into /compiler-rt/lib/tsan/go/
  3. ./buildgo.sh (The test step crashes, but the library works anyways. My guess is that the test does not use a proper Go environment.)
  4. Copy race_linux_amd64.syso to $GOPATH/src/runtime/race/

I'm not sure how to continue from here. Any suggestions on getting that upstream?

@dvyukov

This comment has been minimized.

Show comment
Hide comment
@dvyukov

dvyukov Feb 24, 2017

Member

Here are instructions on how to contribute to sanitizers:
https://github.com/google/sanitizers/wiki/AddressSanitizerHowToContribute

I skimmed through your patch and I think we can upstream something like this. But we need some testing story, otherwise it will break in future.

One good test might be to whitelist all libc symbol dependencies in buildgo.sh. This way we can ensure that we won't silently add a new dependency (ideally in future this list becomes empty).

Member

dvyukov commented Feb 24, 2017

Here are instructions on how to contribute to sanitizers:
https://github.com/google/sanitizers/wiki/AddressSanitizerHowToContribute

I skimmed through your patch and I think we can upstream something like this. But we need some testing story, otherwise it will break in future.

One good test might be to whitelist all libc symbol dependencies in buildgo.sh. This way we can ensure that we won't silently add a new dependency (ideally in future this list becomes empty).

@neelance

This comment has been minimized.

Show comment
Hide comment
@neelance

neelance Feb 24, 2017

Member

@dvyukov I see that you did most of the updates of the syso files. I have some more questions:

  1. How can we test the patch properly on the Go end across all platforms so we don't upstream something stupid?
  2. Can we somehow add Alpine to the platforms being tested?
  3. We need to fix that crash on the test phase of buildgo.sh. I've investigated it a bit, seems like mmap is failing for some shadow memory. My wild guess right now is that it expects a Go memory layout, but the test is a C program. That was as far as I could get without spending a massive amount of time to understand how that thread sanitizer works.
Member

neelance commented Feb 24, 2017

@dvyukov I see that you did most of the updates of the syso files. I have some more questions:

  1. How can we test the patch properly on the Go end across all platforms so we don't upstream something stupid?
  2. Can we somehow add Alpine to the platforms being tested?
  3. We need to fix that crash on the test phase of buildgo.sh. I've investigated it a bit, seems like mmap is failing for some shadow memory. My wild guess right now is that it expects a Go memory layout, but the test is a C program. That was as far as I could get without spending a massive amount of time to understand how that thread sanitizer works.
@dvyukov

This comment has been minimized.

Show comment
Hide comment
@dvyukov

dvyukov Feb 24, 2017

Member

How can we test the patch properly on the Go end across all platforms so we don't upstream something stupid?

I now use golang.org/x/build/cmd/racebuild which builds and tests race runtime for all platforms. But it requires gomote access (I think you need to be a committer for Go repo). If you don't have that access, I will do testing. But test at least on Alpine and on glibc-based Linux.

Can we somehow add Alpine to the platforms being tested?

You need to either run your own Go builder and connect it to dashboard, or we need to setup Alpine linux builder on GCE. @bradfitz, do we want to setup/maintain it?

seems like mmap is failing for some shadow memory. My wild guess right now is that it expects a Go memory layout

It's possible. Does removing -fPIC -fpie from buildgo.sh help? We still need to build the runtime with -pie, but the test itself does not need to be pie.

Member

dvyukov commented Feb 24, 2017

How can we test the patch properly on the Go end across all platforms so we don't upstream something stupid?

I now use golang.org/x/build/cmd/racebuild which builds and tests race runtime for all platforms. But it requires gomote access (I think you need to be a committer for Go repo). If you don't have that access, I will do testing. But test at least on Alpine and on glibc-based Linux.

Can we somehow add Alpine to the platforms being tested?

You need to either run your own Go builder and connect it to dashboard, or we need to setup Alpine linux builder on GCE. @bradfitz, do we want to setup/maintain it?

seems like mmap is failing for some shadow memory. My wild guess right now is that it expects a Go memory layout

It's possible. Does removing -fPIC -fpie from buildgo.sh help? We still need to build the runtime with -pie, but the test itself does not need to be pie.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Feb 24, 2017

Member

#17891 tracks adding an Alpine builder. @jessfraz has a (stalled :)) CL in progress.

Member

bradfitz commented Feb 24, 2017

#17891 tracks adding an Alpine builder. @jessfraz has a (stalled :)) CL in progress.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Apr 25, 2017

Member

We now have Alpine builders, so in theory golang.org/x/build/cmd/racebuild could build an Alpine race binary now.

For now I'm just going to be skipping race tests on Alpine so we don't regress on other things.

Member

bradfitz commented Apr 25, 2017

We now have Alpine builders, so in theory golang.org/x/build/cmd/racebuild could build an Alpine race binary now.

For now I'm just going to be skipping race tests on Alpine so we don't regress on other things.

@gopherbot

This comment has been minimized.

Show comment
Hide comment

CL https://golang.org/cl/41678 mentions this issue.

gopherbot pushed a commit that referenced this issue Apr 25, 2017

cmd/go, cmd/dist: temporarily disable race and PIE internal link test…
…s on Alpine

In an effort to at least understand the complete set of things not
working on Alpine Linux, I've been trying to get the build passing
again, even with tests disabled.

The race detector is broken on Alpine. That is #14481 (and #9918).
So disable those tests for now.

Also, internal linking with PIE doesn't work on Alpine yet.
That is #18243. So disable that test for now.

With this CL, all.bash almost passes. There's some cgo test failing
still, but there's no bug yet, so that can be a separate CL.

Change-Id: I3ffbb0e787ed54cb82f298b6bd5bf3ccfbc82622
Reviewed-on: https://go-review.googlesource.com/41678
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

@tamird tamird referenced this issue in cockroachdb/cockroach Apr 26, 2017

Closed

build: Race build on merge to master broken #15378

benesch added a commit to benesch/cockroach that referenced this issue Apr 26, 2017

build: don't (fail to) build musl race binaries
The race detector doesn't work without glibc due to golang/go#14481.

Fixes #15378.

@slok slok referenced this issue in themotion/ladder May 6, 2017

Merged

Fix race condition on tests #19

@jamesjwarren jamesjwarren referenced this issue in thisissoon/GoCookieCutter May 30, 2017

Merged

Fix: Gitlab test/build jobs #5

AkihiroSuda pushed a commit to AkihiroSuda/linuxkit that referenced this issue Jun 6, 2017

Akihiro Suda
this won't work on alpine
golang/go#14481

Signed-off-by: Akihiro Suda <suda@suda-mbp.osrg.net>

@AkihiroSuda AkihiroSuda referenced this issue in linuxkit/linuxkit Jun 6, 2017

Closed

test: add test-containerd #1906

@mattbostock mattbostock referenced this issue in mattbostock/timbala Jul 16, 2017

Closed

Enable race detection when running tests #58

@mattbostock mattbostock referenced this issue in mattbostock/timbala Jul 17, 2017

Merged

Enable race detection during tests #60

mattbostock added a commit to mattbostock/timbala that referenced this issue Jul 17, 2017

Enable race detection during tests
Enable the Go race detector to help surface race conditions when
running the tests.

The race detector currently depends on libc, so does not work with
Alpine Linux (which uses musl):

golang/go#9918
golang/go#14481

Instead, use the default Go Docker image, which uses the libc-based
Debian Jessie and update the Dockerfiles accordingly.

mattbostock added a commit to mattbostock/timbala that referenced this issue Jul 17, 2017

Enable race detection during tests
Enable the Go race detector to help surface race conditions when
running the tests.

The race detector currently depends on libc, so does not work with
Alpine Linux (which uses musl):

golang/go#9918
golang/go#14481

Instead, use the default Go Docker image, which uses the libc-based
Debian Jessie and update the Dockerfiles accordingly.
@djui

This comment has been minimized.

Show comment
Hide comment
@djui

djui Dec 1, 2017

What is the latest update on this issue?

djui commented Dec 1, 2017

What is the latest update on this issue?

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Dec 1, 2017

Member

@djui, there is no update. Nobody on the Go team is working on this, and nobody elsewhere in the community has posted anything here about it. It's all yours if you want to work on it.

Member

bradfitz commented Dec 1, 2017

@djui, there is no update. Nobody on the Go team is working on this, and nobody elsewhere in the community has posted anything here about it. It's all yours if you want to work on it.

@bradfitz bradfitz added the NeedsFix label Dec 1, 2017

@yosifkit yosifkit referenced this issue in docker-library/golang Mar 20, 2018

Closed

gcc failed #155

@mestery mestery referenced this issue in ligato/networkservicemesh Apr 24, 2018

Merged

Add a build.sh script #26

mestery added a commit to mestery/networkservicemesh that referenced this issue Apr 24, 2018

Remove go test -race
Turns out there is an issue with alpine linux and "go test -race" [1].

[1] golang/go#14481

Signed-off-by: Kyle Mestery <mestery@mestery.com>

@italolelis italolelis referenced this issue in hellofresh/janus May 28, 2018

Merged

Add Timeouts functionality #323

@acroca acroca referenced this issue in kowala-tech/docker-go-dev Jul 5, 2018

Open

Use alpine #4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment