Multirun cannot find bash runfiles

[dimitarvdimitrov]

We're trying to use multirun to do multiple container_push's but can't run the multirun target even for simple binary targets.

Building of the multirun target succeeds but when running it we get

/private/var/tmp/_bazel_dimitrovd/a9507c16f9ff4a741b31cd8f280aebd6/execroot/__main__/bazel-out/darwin-fastbuild/bin/run_all.bash: line 18: /private/var/tmp/_bazel_dimitrovd/a9507c16f9ff4a741b31cd8f280aebd6/external/bazel_tools/tools/bash/runfiles/runfiles.bash: No such file or directory

Although I can see the file exists

lrwxr-xr-x  1 dimitrovd  wheel   121B  2 May 13:26 bazel-out/darwin-fastbuild/bin/run_all.bash.runfiles/bazel_tools/tools/bash/runfiles/runfiles.bash -> /private/var/tmp/_bazel_dimitrovd/a9507c16f9ff4a741b31cd8f280aebd6/external/bazel_tools/tools/bash/runfiles/runfiles.bash

bazel version

Build label: 0.24.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Apr 2 16:32:47 2019 (1554222767)
Build timestamp: 1554222767
Build timestamp as int: 1554222767

Issue is present when using 20cbdb1 of bazel-tools

A workaround was to downgrade to c7e0fd9

I'm posting below some of the relevant files

/BUILD:

load("@com_github_atlassian_bazel_tools//:multirun/def.bzl", "multirun")

multirun(
    name = "run_all",
    commands = [
        "//go/cmd/app:binary"
    ]
)

/go/cmd/app/BUILD:

...
load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")

go_library(
    name = "go_default_library",
    srcs = ["main.go"],
    importpath = "gitlab.local/team/cmd/app",
    visibility = ["//visibility:private"],
)

go_binary(
    name = "binary",
    embed = [":go_default_library"],
    visibility = ["//visibility:public"],
)
...

/WORKSPACE:

...
skylib_version = "0.8.0"

http_archive(
    name = "bazel_skylib",
    type = "tar.gz",
    url = "https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib.{}.tar.gz".format(skylib_version, skylib_version),
    sha256 = "2ef429f5d7ce7111263289644d233707dba35e39696377ebab8b0bc701f7818e",
)

http_archive(
    name = "com_github_atlassian_bazel_tools",
    strip_prefix = "bazel-tools-20cbdb188d18c5470697783258cd2ec2b531b710",
    urls = ["https://github.com/atlassian/bazel-tools/archive/20cbdb188d18c5470697783258cd2ec2b531b710.zip"],
)

load("@com_github_atlassian_bazel_tools//:multirun/deps.bzl", "multirun_dependencies")

multirun_dependencies()

load("@com_github_atlassian_bazel_tools//golangcilint:deps.bzl", "golangcilint_dependencies")

golangcilint_dependencies()
...

/bazel-out/darwin-fastbuild/bin/run_all.bash:

#!/usr/bin/env bash

# --- begin runfiles.bash initialization ---
# Copy-pasted from Bazel's Bash runfiles library (tools/bash/runfiles/runfiles.bash).
set -euo pipefail
if [[ ! -d "${RUNFILES_DIR:-/dev/null}" && ! -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then
  if [[ -f "$0.runfiles_manifest" ]]; then
    export RUNFILES_MANIFEST_FILE="$0.runfiles_manifest"
  elif [[ -f "$0.runfiles/MANIFEST" ]]; then
    export RUNFILES_MANIFEST_FILE="$0.runfiles/MANIFEST"
  elif [[ -f "$0.runfiles/bazel_tools/tools/bash/runfiles/runfiles.bash" ]]; then
    export RUNFILES_DIR="$0.runfiles"
  fi
fi
if [[ -f "${RUNFILES_DIR:-/dev/null}/bazel_tools/tools/bash/runfiles/runfiles.bash" ]]; then
  source "${RUNFILES_DIR}/bazel_tools/tools/bash/runfiles/runfiles.bash"
elif [[ -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then
  source "$(grep -m1 "^bazel_tools/tools/bash/runfiles/runfiles.bash "             "$RUNFILES_MANIFEST_FILE" | cut -d ' ' -f 2-)"
else
  echo >&2 "ERROR: cannot find @bazel_tools//tools/bash/runfiles:runfiles.bash"
  exit 1
fi
# --- end runfiles.bash initialization ---

# Export RUNFILES_* envvars (and a couple more) for subprocesses.
runfiles_export_envvars

echo Running '//go/cmd/app:binary'
./'go/cmd/app/darwin_amd64_stripped/binary' $@

Let me know if you need anything else shared.

[ash2k]

@dimitarvdimitrov Would be great to find out which commit actually broke it. Could you see if e31d79d works

@laszlocsomor maybe you have an idea?

[dimitarvdimitrov]

@ash2k Same issue there as well. After a bit of investigation, found that there is an extra space in the grep regex at

bazel-tools/multirun/def.bzl

Line 20 in 20cbdb1

source "$(grep -m1 "^bazel_tools/tools/bash/runfiles/runfiles.bash " \

Removing it solved the problem.

Should I submit a PR for that?

[laszlocsomor]

@dimitarvdimitrov : That extra space should be there: https://github.com/bazelbuild/bazel/blob/9eb2ef0d1feea9a08161ea0f8c16e3ac0a07064d/tools/bash/runfiles/runfiles.bash#L53

Let me try to repro this error locally.

[laszlocsomor]

I fail to repro this on Linux. Do you only see this on macOS?

What's the output of:

grep 'bazel_tools/tools/bash/runfiles/runfiles.bash' bazel-out/darwin-fastbuild/bin/run_all.bash.runfiles_manifest

?

[dimitarvdimitrov]

@laszlocsomor Managed to dig up the problem; it is because of my GREP_OPTIONS=--color=always.

Thanks for the help and sorry for bothering you guys with it.

Closing as it isn't an issue with bazel-tools

[laszlocsomor]

Thanks for figuring that out! I wasn't aware GREP_OPTIONS existed.
Maybe I should make the script runfiles.bash init code ignore it.

[ash2k]

@laszlocsomor I think the only safe/deterministic solution would be to completely shield grep from all environment variables using env grep .... Same with any other programs in that script (if any) - they can be influenced by an environment variable, if not today, maybe in the future.