Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Buildkit SIGSEGV nil pointer exception #2286

Open
3 tasks
r-LaForge opened this issue Oct 18, 2022 · 2 comments
Open
3 tasks

Buildkit SIGSEGV nil pointer exception #2286

r-LaForge opened this issue Oct 18, 2022 · 2 comments
Labels
type:bug Something isn't working

Comments

@r-LaForge
Copy link
Contributor

r-LaForge commented Oct 18, 2022 •
edited

Slack conversation: https://earthlycommunity.slack.com/archives/C01DL2928RM/p1666094497824899
A nil pointer exception is sometimes encountered when running an earthly target.

The behaviour has been seen both when running a remote docker and a local docker in GitHub Actions.
Earthfile and Buildkit logs are included below.
The line where the nil pointer exception is seen is here: https://github.com/moby/buildkit/blob/master/solver/jobs.go#L525

Resolution

  • Determine the underlying cause of the issue.
  • Update buildkit as we are a few months behind upstream (in case it was already resolved).
  • Tidy the error message when failing to unmarshal the response from docker version.

Earthfile

Minimum

VERSION 0.6

test-case:
  FROM debian
  RUN --push mkdir -p artifact
  SAVE ARTIFACT artifact AS LOCAL test-output/artifact

VERSION 0.6

all:
  BUILD +push-artifacts

test-preparation:
  FROM debian

  RUN mkdir /work
  WORKDIR /work

  RUN apt-get update -y && \
    apt-get install -y \
      puppet \
      puppet-lint \
      bundler \
      libsodium-dev \
      curl \
      git

  COPY Gemfile .
  RUN bundle install

  # Install gsutils
  RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
   curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg  add - && \
   apt-get update -y && apt-get install -y google-cloud-sdk

  COPY --dir \
    bootstrap \
    build \
    common_data_all_envs \
    environments \
    modules \
    spec \
    test \
    kitchen-google.yml \
    verifier.sh \
    test_run.sh \
    Gemfile \
    .

  COPY . .

  RUN ls -alh .
  RUN du -sh *
  RUN test/test_setup.sh
  RUN test/syntax_check.sh

push-artifacts:
  FROM +test-preparation
  ARG SANITIZED_BRANCH
  ARG GOOGLE_APPLICATION_CREDENTIALS=google_credentials.json
  RUN --mount type=secret,target=${GOOGLE_APPLICATION_CREDENTIALS},id=+secrets/gcloud \
    gcloud auth activate-service-account --key-file=${GOOGLE_APPLICATION_CREDENTIALS}
  IF [ "${SANITIZED_BRANCH}" = "master" ]
    RUN --push build/build_artifacts.sh
  ELSE
    RUN --push build/build_artifacts-feature-branch.sh
  END
  SAVE ARTIFACT artifact AS LOCAL test-output/artifact

Earthly Logs

Two different scenarios:

00:39:40.549  Total       	38m59.024992861s
00:39:40.549  Total (real)	38m42.140056269s
00:39:40.549  Share your logs with an Earthly account (experimental)! Register for one at https://ci.earthly.dev/.
00:39:40.549  Error stack trace:
00:39:40.549  github.com/moby/buildkit/client.(*Client).solve.func4
00:39:40.549  	/go/pkg/mod/github.com/earthly/buildkit@v0.0.1-0.20220913002628-9c256c19c539/client/solve.go:300
00:39:40.549  golang.org/x/sync/errgroup.(*Group).Go.func1
00:39:40.549  	/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
00:39:40.549  runtime.goexit
00:39:40.549  	/usr/local/go/src/runtime/asm_amd64.s:1581
00:39:40.549  Error: build target: build push: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF
00:39:40.809

00:01:11.272              frontend | Running command: docker info
00:01:11.530              frontend | Running command: docker info --format={{.SecurityOptions}}
00:01:11.530              frontend | docker-shell frontend initialized.
00:01:11.787              frontend | Running command: docker container inspect earthly-buildkitd
00:01:11.787             buildkitd | Starting buildkit daemon as a docker container (earthly-buildkitd)...
00:01:11.787              frontend | Running command: docker container inspect earthly-buildkitd
00:01:11.787              frontend | Running command: docker rm earthly-buildkitd
00:01:11.787              frontend | Running command: docker image inspect earthly/buildkitd:v0.6.27
00:01:11.787              frontend | Running command: docker version --format={{json .}}
00:01:11.787              frontend | Running command: docker run --privileged --env BUILDKIT_TLS_ENABLED=false --env BUILDKIT_MAX_PARALLELISM=20 --env BUILDKIT_DEBUG=false --env BUILDKIT_TCP_TRANSPORT_ENABLED=false --label dev.earthly.settingshash=2593e4a9ed74e9f --mount type=volume,source=earthly-cache,dst=/tmp/earthly --publish 127.0.0.1:8373:8373/tcp --publish 127.0.0.1:8371:8371/tcp -d --name earthly-buildkitd earthly/buildkitd:v0.6.27
00:01:12.046  Share your logs with an Earthly account (experimental)! Register for one at https://ci.earthly.dev/.
00:01:12.046  Error stack trace:
00:01:12.046  github.com/earthly/earthly/util/containerutil.(*dockerShellFrontend).Information
00:01:12.046  	/earthly/util/containerutil/docker.go:102
00:01:12.046  github.com/earthly/earthly/util/containerutil.(*shellFrontend).supportsPlatform
00:01:12.046  	/earthly/util/containerutil/shell_shared.go:343
00:01:12.046  github.com/earthly/earthly/util/containerutil.(*shellFrontend).ContainerRun
00:01:12.046  	/earthly/util/containerutil/shell_shared.go:197
00:01:12.046  github.com/earthly/earthly/buildkitd.Start
00:01:12.046  	/earthly/buildkitd/buildkitd.go:466
00:01:12.046  github.com/earthly/earthly/buildkitd.MaybeStart
00:01:12.046  	/earthly/buildkitd/buildkitd.go:178
00:01:12.046  github.com/earthly/earthly/buildkitd.NewClient
00:01:12.046  	/earthly/buildkitd/buildkitd.go:76
00:01:12.046  main.(*earthlyApp).actionBuildImp
00:01:12.046  	/earthly/cmd/earthly/build_cmd.go:221
00:01:12.046  main.(*earthlyApp).actionBuild
00:01:12.046  	/earthly/cmd/earthly/build_cmd.go:78
00:01:12.046  github.com/urfave/cli/v2.(*App).RunContext
00:01:12.046  	/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:322
00:01:12.046  main.(*earthlyApp).run
00:01:12.046  	/earthly/cmd/earthly/main.go:535
00:01:12.046  main.main
00:01:12.046  	/earthly/cmd/earthly/main.go:268
00:01:12.046  runtime.main
00:01:12.046  	/usr/local/go/src/runtime/proc.go:255
00:01:12.047  runtime.goexit
00:01:12.047  	/usr/local/go/src/runtime/asm_amd64.s:1581
00:01:12.047  Error: build new buildkitd client: maybe start buildkitd: start: could not start buildkit: 1 error occurred:
00:01:12.047  	* failed to get platform information: failed to parse docker version output: invalid character 'W' after top-level value

Buildkit logs

docker logs -f 9cdf74323683
starting earthly-buildkit with EARTHLY_GIT_HASH=eab653d7fd99146a1f72df1ddaebd64e90b24046 BUILDKIT_BASE_IMAGE=github.com/earthly/buildkit:9c256c19c5397aee1ba0379590d685c2e87ff17c+build
Autodetecting iptables
Could not find an ip_tables module; falling back to heuristics.
# Warning: iptables-legacy tables present, use iptables-legacy to see them
iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument

Detected iptables-legacy by output length (4 >= 0)
BUILDKIT_ROOT_DIR=/tmp/earthly/buildkit
CACHE_SIZE_MB=0
BUILDKIT_MAX_PARALLELISM=20
BUILDKIT_LOCAL_REGISTRY_LISTEN_PORT=8371
EARTHLY_ADDITIONAL_BUILDKIT_CONFIG=
CNI_MTU=1460

======== CNI config ==========
{
	"cniVersion": "0.3.0",
	"name": "buildkitbuild",
	"type": "bridge",
	"bridge": "cni0",
	"isGateway": true,
	"ipMasq": true,
	"mtu": 1460,
	"ipam": {
		"type": "host-local",
		"subnet": "172.30.0.0/16",
		"routes": [
			{ "dst": "0.0.0.0/0" }
		]
	}
}
======== End CNI config ==========

======== Buildkitd config ==========
debug = false
root = "/tmp/earthly/buildkit"
insecure-entitlements = [ "security.insecure" ]





[worker.oci]
  enabled = true
  snapshotter = "auto"
  max-parallelism = 20
  gc = true
  networkMode = "cni"
  cniBinaryPath = "/usr/libexec/cni"
  cniConfigPath = "/etc/cni/cni-conf.json"



======== End buildkitd config ==========
Detected container architecture is x86_64
time="2022-10-18T14:11:33Z" level=info msg="auto snapshotter: using overlayfs"
time="2022-10-18T14:11:34Z" level=info msg="found worker \"jpvyfp6dvl9w4iq3qyzo9gqh6\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:9cdf74323683 org.mobyproject.buildkit.worker.network:cni org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/386]"
time="2022-10-18T14:11:34Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2022-10-18T14:11:34Z" level=info msg="found 1 workers, default=\"jpvyfp6dvl9w4iq3qyzo9gqh6\""
time="2022-10-18T14:11:34Z" level=warning msg="currently, only the default worker can be used."
time="2022-10-18T14:11:34Z" level=info msg="Starting local registry for outputs on port 8371"
time="2022-10-18T14:11:34Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.18.7
time="2022-10-18T14:11:34Z" level=info msg="redis not configured" go.version=go1.18.7
time="2022-10-18T14:11:34Z" level=info msg="Starting upload purge in 17m0s" go.version=go1.18.7
time="2022-10-18T14:11:34Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=6a4fa887-c36e-4def-969a-73758a7574f9 http.request.method=GET http.request.remoteaddr="169.254.123.1:36812" http.request.uri=/v2/ http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.13722ms http.response.status=200 http.response.written=2
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=aa8f5e4b-a5e9-413f-9306-949b5fa7c660 http.request.method=GET http.request.remoteaddr="169.254.123.1:36834" http.request.uri=/v2/ http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.052208ms http.response.status=200 http.response.written=2
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=51b83083-8c71-4e1d-997c-019b3cd1b834 http.request.method=HEAD http.request.remoteaddr="169.254.123.1:36848" http.request.uri=/v2/sess-eoa72icsi81s3b3yx09ffbhoc/pullping/manifests/img-0 http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=1.246212ms http.response.status=200 http.response.written=2630
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=2812c7de-d758-45cb-9622-98c865953db2 http.request.method=HEAD http.request.remoteaddr="169.254.123.1:36856" http.request.uri=/v2/sess-eoa72icsi81s3b3yx09ffbhoc/pullping/manifests/img-1 http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=1.544327ms http.response.status=200 http.response.written=2630
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=aee760e5-4c80-487e-81b8-6ad8c7888a65 http.request.method=GET http.request.remoteaddr="169.254.123.1:36872" http.request.uri="/v2/sess-eoa72icsi81s3b3yx09ffbhoc/pullping/manifests/sha256:84fcb34e0706725e180b2a43b1120daa369712fb7c050b61a18069da396c59b8" http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=1.519569ms http.response.status=200 http.response.written=2630
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=4390b7f3-17fd-430c-b254-2ccc622c1b48 http.request.method=GET http.request.remoteaddr="169.254.123.1:36880" http.request.uri="/v2/sess-eoa72icsi81s3b3yx09ffbhoc/pullping/manifests/sha256:84fcb34e0706725e180b2a43b1120daa369712fb7c050b61a18069da396c59b8" http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=1.584009ms http.response.status=200 http.response.written=2630
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=8f04d90b-b600-42a3-ba68-e0c26e484322 http.request.method=GET http.request.remoteaddr="169.254.123.1:36888" http.request.uri="/v2/sess-eoa72icsi81s3b3yx09ffbhoc/pullping/blobs/sha256:96d8e2195cec7e6355c9967f36047bcca69a58c8355052bc96ae7ed623241ec9" http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype=application/octet-stream http.response.duration=1.772438ms http.response.status=200 http.response.written=4570
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=857d54d4-6779-43b9-a4ec-305ac74680de http.request.method=GET http.request.remoteaddr="169.254.123.1:36890" http.request.uri="/v2/sess-eoa72icsi81s3b3yx09ffbhoc/pullping/blobs/sha256:96d8e2195cec7e6355c9967f36047bcca69a58c8355052bc96ae7ed623241ec9" http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype=application/octet-stream http.response.duration=6.498127ms http.response.status=200 http.response.written=4570
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=7f7a02d1-8f85-4de9-aa81-099fe41765d7 http.request.method=GET http.request.remoteaddr="169.254.123.1:36906" http.request.uri="/v2/sess-eoa72icsi81s3b3yx09ffbhoc/pullping/blobs/sha256:768f922b4ba9624809b43d60ad66e383fba6fea49a0daccb68edd96c6b41687a" http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype=application/octet-stream http.response.duration=10.835302ms http.response.status=200 http.response.written=1412228
time="2022-10-18T14:13:19Z" level=info msg="response completed" go.version=go1.18.7 http.request.host="127.0.0.1:8371" http.request.id=f3a55667-cdf4-41be-82b8-d2d909c4cd9d http.request.method=GET http.request.remoteaddr="169.254.123.1:36900" http.request.uri="/v2/sess-eoa72icsi81s3b3yx09ffbhoc/pullping/blobs/sha256:10fd9cdcf7d6c04b6cca60474803150296c4da18ec46a27ced72dd5241320954" http.request.useragent="docker/20.10.3 go/go1.13.5 git-commit/46229ca1d8 kernel/5.4.202+ os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.19 \\(linux\\))" http.response.contenttype=application/octet-stream http.response.duration=18.639061ms http.response.status=200 http.response.written=349152






panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0xc68b69]

goroutine 66169 [running]:
github.com/moby/buildkit/solver.(*Job).walkBuildSources(0xc007fe0400, {0x1aa7438, 0xc015187980}, {0x47?, {0x1aa9e50?, 0xc015207240?}}, 0xc014e14b70)
	/src/solver/jobs.go:526 +0x169
github.com/moby/buildkit/solver.(*Job).walkBuildSources(0xc007fe0400, {0x1aa7438, 0xc015187980}, {0x0?, {0x1aa9e50?, 0xc015207280?}}, 0xc014e14b70)
	/src/solver/jobs.go:534 +0x1dc
github.com/moby/buildkit/solver.(*Job).Build(0xc007fe0400, {0x1aa7438, 0xc015187980}, {0x0, {0x1aa9ed0?, 0xc00a2a7700?}})
	/src/solver/jobs.go:519 +0x3af
github.com/moby/buildkit/solver/llbsolver.(*llbBridge).loadResult(0xc014572320, {0x1aa7438?, 0xc015187980}, 0x28?, {0x0, 0x0, 0x40d665?})
	/src/solver/llbsolver/bridge.go:153 +0x47c
github.com/moby/buildkit/solver/llbsolver.newResultProxy.func1({0x1aa7438?, 0xc015187980?})
	/src/solver/llbsolver/bridge.go:278 +0x65
github.com/moby/buildkit/solver/llbsolver.(*resultProxy).Result.func2({0x1aa7438, 0xc015187980})
	/src/solver/llbsolver/bridge.go:359 +0x150
github.com/moby/buildkit/util/flightcontrol.(*call).run(0xc000cb8a20)
	/src/util/flightcontrol/flightcontrol.go:121 +0x5e
sync.(*Once).doSlow(0xc015194808?, 0xc0076effd0?)
	/usr/local/go/src/sync/once.go:68 +0xc2
sync.(*Once).Do(0xc0161b9620?, 0xc0076eff98?)
	/usr/local/go/src/sync/once.go:59 +0x1f
created by github.com/moby/buildkit/util/flightcontrol.(*call).wait
	/src/util/flightcontrol/flightcontrol.go:164 +0x44a
@r-LaForge r-LaForge added the type:bug Something isn't working label Oct 18, 2022
@ingwarsw
Copy link
Contributor

I was able to make it a bit simpler

VERSION 0.6

test-case:
  FROM debian
  RUN --push mkdir -p artifact
  SAVE ARTIFACT artifact AS LOCAL test-output/artifact

And I was even able to reproduce it on arm mac.

@r-LaForge
Copy link
Contributor Author

The fix will be automatically included in VERSION 0.7, but there is a workaround for now:
add --wait-block to the version line. VERSION --wait-block 0.6
Internally, this changes the behaviour of our --push implementation, which seems to resolve the issue.
This is this Earthfile passes now.
With that feature a new BUILT_IN arg is also exposed (not documented yet) that is set to "true" if --push is specified:

VERSION --wait-block 0.6

push:
  FROM alpine
  RUN echo "in the push target"

test:
  FROM debian
  ARG EARTHLY_PUSH
  IF [ "$EARTHLY_PUSH" = "true" ]
    BUILD +push
  END

@r-LaForge r-LaForge mentioned this issue Oct 20, 2022
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug Something isn't working
Projects
core
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ingwarsw @r-LaForge