bazel run does not work with remote caching and remote_download_minimal

[ash2k]

Description of the problem:

bazel run //target seem to not work with remote caching and --remote_download_minimal flag.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

Linux, x86_64

What's the output of bazel info release?

release 3.4.1

Have you found anything relevant by searching the web?

#6862 (comment)
#6862 (comment)

Any other information, logs, or outputs that you want to share?

I initially thought that this is a gazelle issue so reported it there, please see bazelbuild/bazel-gazelle#884. But both gazelle and rules_docker when invoked via bazel run fail in a similar way:

Gazelle in the failing build: (gazelle was modified with -x in internal/gazelle.bash.in for verbosity).

(02:37:58) SUBCOMMAND: # @bazel_gazelle//cmd/gazelle:gazelle [action 'GoLink external/bazel_gazelle/cmd/gazelle/gazelle_/gazelle [for host]', configuration: e0f8bdaa6b837c72d742f4b6729a427d9d3357a25a5d2b75f2dfb36fd12e8d1d, execution platform: @local_config_platform//:host]
(cd /root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/execroot/gitlab_k8s_agent && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/usr/local/bin:/bin \
  bazel-out/host/bin/external/go_sdk/builder link -sdk external/go_sdk -installsuffix linux_amd64 -arc '@bazel_gazelle//internal/wspace:wspace=github.com/bazelbuild/bazel-gazelle/internal/wspace=bazel-out/host/bin/external/bazel_gazelle/internal/wspace/wspace.a' -arc '@bazel_gazelle//pathtools:pathtools=github.com/bazelbuild/bazel-gazelle/pathtools=bazel-out/host/bin/external/bazel_gazelle/pathtools/pathtools.a' -arc '@bazel_gazelle//label:label=github.com/bazelbuild/bazel-gazelle/label=bazel-out/host/bin/external/bazel_gazelle/label/label.a' -arc '@com_github_bazelbuild_buildtools//tables:go_default_library=github.com/bazelbuild/buildtools/tables=bazel-out/host/bin/external/com_github_bazelbuild_buildtools/tables/go_default_library.a' -arc '@com_github_bazelbuild_buildtools//build:go_default_library=github.com/bazelbuild/buildtools/build=bazel-out/host/bin/external/com_github_bazelbuild_buildtools/build/go_default_library.a' -arc '@bazel_gazelle//rule:rule=github.com/bazelbuild/bazel-gazelle/rule=bazel-out/host/bin/external/bazel_gazelle/rule/rule.a' -arc '@bazel_gazelle//config:config=github.com/bazelbuild/bazel-gazelle/config=bazel-out/host/bin/external/bazel_gazelle/config/config.a' -arc '@org_golang_x_tools//go/vcs:vcs=golang.org/x/tools/go/vcs=bazel-out/host/bin/external/org_golang_x_tools/go/vcs/vcs.a' -arc '@bazel_gazelle//repo:repo=github.com/bazelbuild/bazel-gazelle/repo=bazel-out/host/bin/external/bazel_gazelle/repo/repo.a' -arc '@bazel_gazelle//resolve:resolve=github.com/bazelbuild/bazel-gazelle/resolve=bazel-out/host/bin/external/bazel_gazelle/resolve/resolve.a' -arc '@bazel_gazelle//language:language=github.com/bazelbuild/bazel-gazelle/language=bazel-out/host/bin/external/bazel_gazelle/language/language.a' -arc '@bazel_gazelle//language/proto:proto=github.com/bazelbuild/bazel-gazelle/language/proto=bazel-out/host/bin/external/bazel_gazelle/language/proto/proto.a' -arc '@bazel_gazelle//flag:flag=github.com/bazelbuild/bazel-gazelle/flag=bazel-out/host/bin/external/bazel_gazelle/flag/flag.a' -arc '@bazel_gazelle//internal/version:version=github.com/bazelbuild/bazel-gazelle/internal/version=bazel-out/host/bin/external/bazel_gazelle/internal/version/version.a' -arc '@com_github_pelletier_go_toml//:go-toml=github.com/pelletier/go-toml=bazel-out/host/bin/external/com_github_pelletier_go_toml/go-toml.a' -arc '@org_golang_x_sync//errgroup:errgroup=golang.org/x/sync/errgroup=bazel-out/host/bin/external/org_golang_x_sync/errgroup/errgroup.a' -arc '@bazel_gazelle//language/go:go=github.com/bazelbuild/bazel-gazelle/language/go=bazel-out/host/bin/external/bazel_gazelle/language/go/go.a' -arc '@bazel_gazelle//merger:merger=github.com/bazelbuild/bazel-gazelle/merger=bazel-out/host/bin/external/bazel_gazelle/merger/merger.a' -arc '@com_github_bmatcuk_doublestar//:doublestar=github.com/bmatcuk/doublestar=bazel-out/host/bin/external/com_github_bmatcuk_doublestar/doublestar.a' -arc '@bazel_gazelle//walk:walk=github.com/bazelbuild/bazel-gazelle/walk=bazel-out/host/bin/external/bazel_gazelle/walk/walk.a' -arc '@com_github_pmezard_go_difflib//difflib:difflib=github.com/pmezard/go-difflib/difflib=bazel-out/host/bin/external/com_github_pmezard_go_difflib/difflib/difflib.a' -package_list bazel-out/host/bin/external/go_sdk/packages.txt -o bazel-out/host/bin/external/bazel_gazelle/cmd/gazelle/gazelle_/gazelle -main bazel-out/host/bin/external/bazel_gazelle/cmd/gazelle/gazelle.a -p cmd/gazelle -- -extld /usr/local/bin/clang '-buildid=redacted' -extldflags '-fuse-ld=/usr/bin/ld.gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/local/bin -lm')
Target //:gazelle up-to-date:
  bazel-bin/gazelle-runner.bash
  bazel-bin/gazelle
(02:37:58) INFO: Elapsed time: 54.314s, Critical Path: 1.78s
(02:37:58) INFO: 36 processes: 36 remote cache hit.
(02:37:58) INFO: Build completed successfully, 45 total actions
(02:37:58) INFO: Running command line: bazel-bin/gazelle -bazel_run
(02:37:58) INFO: Build completed successfully, 45 total actions
+ RUNNER_LABEL=//:gazelle-runner
+ GAZELLE_SHORT_PATH=../bazel_gazelle/cmd/gazelle/gazelle_/gazelle
+ GAZELLE_LABEL=@bazel_gazelle//cmd/gazelle:gazelle
+ ARGS=('update' '-mode' 'fix' '-go_prefix' 'gitlab.com/gitlab-org/cluster-integration/gitlab-agent')
+ GOTOOL=external/go_sdk/bin/go
+ is_bazel_run=false
+ '[' -bazel_run = -bazel_run ']'
+ is_bazel_run=true
+ shift
+ '[' '' = -args ']'
+ '[' 0 -ne 0 ']'
+ '[' true = true ']'
+ set_goroot
++ find_runfile external/go_sdk/bin/go
++ local runfile=external/go_sdk/bin/go
++ '[' -f external/go_sdk/bin/go ']'
++ readlink external/go_sdk/bin/go
++ return
+ local gotool=/root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/external/go_sdk/bin/go
+ '[' -z /root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/external/go_sdk/bin/go ']'
+++ dirname /root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/external/go_sdk/bin/go
++ cd /root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/external/go_sdk/bin/..
++ pwd
+ export GOROOT=/root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/external/go_sdk
+ GOROOT=/root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/external/go_sdk
+ type cygpath
++ find_runfile ../bazel_gazelle/cmd/gazelle/gazelle_/gazelle
++ local runfile=../bazel_gazelle/cmd/gazelle/gazelle_/gazelle
++ '[' -f ../bazel_gazelle/cmd/gazelle/gazelle_/gazelle ']'
+++ echo ../bazel_gazelle/cmd/gazelle/gazelle_/gazelle
+++ sed -e 's!^\(\.\./\|external/\)!!'
++ runfile=bazel_gazelle/cmd/gazelle/gazelle_/gazelle
++ grep -q '^bazel_gazelle/cmd/gazelle/gazelle_/gazelle' MANIFEST
grep: MANIFEST: No such file or directory
+ gazelle_short_path=
+ '[' -z '' ']'
+ echo 'error: could not locate gazelle binary'
error: could not locate gazelle binary
+ exit 1

rules_docker in the failing build:

(02:40:03) SUBCOMMAND: # //cmd/agentk:push_docker_tag [action 'ContainerPushDigest cmd/agentk/push_docker_tag.digest', configuration: 905714fd76cef8b2a3b3c0d6dc4474c63bc6192315297b234f69fda2bc05dacb, execution platform: @local_config_platform//:host]
(cd /root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/execroot/gitlab_k8s_agent && \
  exec env - \
  bazel-out/host/bin/external/io_bazel_rules_docker/container/go/cmd/digester/digester_/digester '--config=bazel-out/k8-fastbuild/bin/cmd/agentk/container.0.config' '--layer=external/go_image_static/image/000.tar.gz,bazel-out/k8-fastbuild/bin/external/go_image_static/image/000.tar.gz.nogz,bazel-out/k8-fastbuild/bin/external/go_image_static/image/000.tar.gz.sha256,bazel-out/k8-fastbuild/bin/external/go_image_static/image/000.tar.gz.nogz.sha256' '--layer=bazel-out/k8-fastbuild/bin/cmd/agentk/container-layer.tar.gz,bazel-out/k8-fastbuild/bin/cmd/agentk/container-layer.tar,bazel-out/k8-fastbuild/bin/cmd/agentk/container-layer.tar.gz.sha256,bazel-out/k8-fastbuild/bin/cmd/agentk/container-layer.tar.sha256' '--manifest=bazel-out/k8-fastbuild/bin/cmd/agentk/container.0.manifest' --dst bazel-out/k8-fastbuild/bin/cmd/agentk/push_docker_tag.digest --format Docker)
Target //cmd/agentk:push_docker_tag up-to-date:
  bazel-bin/cmd/agentk/push_docker_tag.digest
  bazel-bin/cmd/agentk/push_docker_tag
(02:40:04) INFO: Elapsed time: 181.350s, Critical Path: 5.19s
(02:40:04) INFO: 620 processes: 620 remote cache hit.
(02:40:04) INFO: Build completed successfully, 633 total actions
(02:40:04) INFO: Running command line: bazel-bin/cmd/agentk/push_docker_tag
(02:40:04) INFO: Build completed successfully, 633 total actions
/root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/execroot/gitlab_k8s_agent/bazel-out/k8-fastbuild/bin/cmd/agentk/push_docker_tag: line 31: /root/.cache/bazel/_bazel_root/9ca8268df963885560ea55034139bea5/execroot/gitlab_k8s_agent/bazel-out/k8-fastbuild/bin/cmd/agentk/push_docker_tag.runfiles/gitlab_k8s_agent/../io_bazel_rules_docker/container/go/cmd/pusher/pusher_/pusher: No such file or directory

Both work fine with --remote_download_toplevel.

[allada]

I suspect the problem is that you need the runfiles and minimal does not download runfiles by default.

Try adding:
--build_runfile_links to the end of the cmdline.

[sitaktif]

Dupe of #11920?

[ash2k]

I'm closing this as it's a duplicate indeed.