Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: mod vendor fails with unlinkat error #38998

Closed
bljusruk opened this issue May 11, 2020 · 9 comments
Closed

cmd/go: mod vendor fails with unlinkat error #38998

bljusruk opened this issue May 11, 2020 · 9 comments

Comments

@bljusruk
Copy link

@bljusruk bljusruk commented May 11, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.10 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build266767394=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I'm trying to build blackbox_exporter within Docker container. I have successfully built it on my CentOS 7 system, now I'm trying to build it in Docker container.

My Dockerfile is:

FROM golang:1.13-alpine AS go

RUN mkdir -p /app/blackbox-exporter

WORKDIR /app/blackbox-exporter

RUN apk add --update --no-cache git curl make gcc libc-dev

ENV PATH $PATH:/go/bin

RUN git clone https://github.com/prometheus/blackbox_exporter.git /app/blackbox-exporter

RUN make

What did you expect to see?

I expect to see no errors when building blackbox_exporter in Docker container.

What did you see instead?

Instead I get the following error:

>> running check for unused packages in vendor/
GO111MODULE=on go mod vendor
go mod vendor: unlinkat /app/blackbox-exporter/vendor/github.com/alecthomas/template/parse: directory not empty
make: *** [/app/blackbox-exporter/Makefile.common:204: common-unused] Error 1
The command '/bin/sh -c make -f /app/blackbox-exporter/blackbox-exporter-makefile' returned a non-zero code: 2

@toothrot toothrot changed the title 'go mod vendor' fails with unlinkat error cmd/go: mod vendor fails with unlinkat error May 11, 2020
@toothrot
Copy link
Contributor

@toothrot toothrot commented May 11, 2020

Thanks for filing. What is the docker command you're using to build? My guess is that you're mounting a local module cache into the container, which is readonly. (see #27161)

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented May 11, 2020

Couldn't reproduce this with Docker on macOS.

Could you give a more minimal example that reproduces the problem? There's a moderately sized Makefile here, and I don't know whether it's doing something that might cause issues.

Is there any chance that more than one instance of go mod vendor is run concurrently?

@bljusruk
Copy link
Author

@bljusruk bljusruk commented May 12, 2020

@toothrot This is the command I'm using to build:
docker build --tag test_img_2 /home/mirov/workspace/sa20sre/blackbox-exporter

@bljusruk
Copy link
Author

@bljusruk bljusruk commented May 12, 2020

@jayconrod I've looked into Makefile.common and from what I see I don't think there is more than one instance of go mod vendor running when I start the build. I have to admit I don't quite understand you when you say 'Could you give a more minimal example that reproduces the problem?'. The make file I use is the one that's already there as it came with go sources. I don't want to mess with it.

@bljusruk
Copy link
Author

@bljusruk bljusruk commented May 12, 2020

@toothroot @jayconrod Temporarily I'm avoiding this problem by removing 'unused' flag in Makefile.common. I have replaced the original line in Makefile.common:

common-all: precheck style check_license lint unused build test

with

common-all: precheck style check_license lint build test

so that 'go mod vendor' is not getting executed.

I did it by putting this line in my Docker file (before calling make):

RUN sed -i 's/precheck style check_license lint unused build test/precheck style check_license lint build test/g' Makefile.common

Now the make goes on without 'go mod vendor' error about directory not being empty.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented May 12, 2020

@jayconrod I've looked into Makefile.common and from what I see I don't think there is more than one instance of go mod vendor running when I start the build. I have to admit I don't quite understand you when you say 'Could you give a more minimal example that reproduces the problem?'. The make file I use is the one that's already there as it came with go sources. I don't want to mess with it.

I'm wondering if there's a way to reproduce this using a few go commands without the Makefile. There's a lot going on in the Makefile, and it's not really an easy language to debug.

From the error message, it sounds like something is writing files in the vendor directory at the same time that go mod vendor runs. go mod vendor starts by calling os.RemoveAll on the vendor directory, and that would fail if it were racing with another process that's adding files. The other process could be another instance of go mod vendor. If this indeed the case, I'm not sure there's much we can do.

I tried this on a couple systems, but I still couldn't reproduce the failure you're seeing. It does however fail with:

>> running all tests
GO111MODULE=on go test -race  -mod=vendor ./...
# runtime/race
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::GetArgv()':
gotsan.cc:(.text+0x4183): undefined reference to `__libc_stack_end'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::ReExec()':
gotsan.cc:(.text+0x9797): undefined reference to `__libc_stack_end'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::InternalAlloc(unsigned long, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<__sanitizer::AP32> >*, unsigned long)':
gotsan.cc:(.text+0xaac1): undefined reference to `__libc_malloc'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::InternalRealloc(void*, unsigned long, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<__sanitizer::AP32> >*)':
gotsan.cc:(.text+0xca20): undefined reference to `__libc_realloc'
/usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: race_linux_amd64.syso: in function `__sanitizer::InternalFree(void*, __sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator32<__sanitizer::AP32> >*)':
gotsan.cc:(.text+0x66e8): undefined reference to `__libc_free'
collect2: error: ld returned 1 exit status
@bljusruk
Copy link
Author

@bljusruk bljusruk commented May 12, 2020

@jayconrod Thanks for answering. Regarding the race error you mentioned, I've also bumped into it. I have worked around it by removing - race flag frm the Makefile.common file. I know it's not the solution but it enabled me to go on with the build. The 'go mod vendor' problem I avoided by removing the flag unused from the Makefile.common file. Again, not the solution but after that I was able to build blackbox_exporter.

@bcmills
Copy link
Member

@bcmills bcmills commented May 12, 2020

The link error looks like #14481, fixed in release-branch.go1.14.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented May 27, 2020

Closing since we haven't been able to reproduce this, and there doesn't seem to be a way to move forward.

@jayconrod jayconrod closed this May 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.