Runfiles: support paths with spaces

[laszlocsomor]

Update: this bug no longer claims to be Windows-specific, see #4327 (comment)

---

Fix this TODO:

bazel/src/main/java/com/google/devtools/build/lib/windows/runfiles/WindowsRunfiles.java

Lines 54 to 55 in 7b423cc

	// TODO(bazel-team): This is buggy when the path contains spaces, we should fix the
	// manifest format.

[izzyleung]

Hmm, this problem also applies to other platforms as well, say macOS, when running bazel build //... on a workspace living under a directory with name containing space(s), bael will tell me:

ERROR: bazel does not currently work properly from paths containing spaces (com.google.devtools.build.lib.runtime.BlazeWorkspace@5dc63968).

[laszlocsomor]

@izzyleung thanks for the info! In that case I think P2 or even P3 is safe. I thought this worked on other platforms.

[laszlocsomor]

Update: I haven't looked into fixing this issue and I'm most likely not going to for the next couple months.

[laszlocsomor]

Dropping priority. Spaces aren't well supported on Linux either. It'd be nice if they were fully supported, but this doesn't seem like a critical feature.

Demo time:

Directory structure:

  $ find .
.
./WORKSPACE
./sub
./sub/a b
./sub/a b/bar
./sub/a b/bar/x.txt
./bin.sh
./BUILD

BUILD file:

sh_test(
    name = "bin",
    srcs = ["bin.sh"],
    data = glob(["sub/**"]),
)

genrule(
    name = "gen",
    outs = ["gen.txt"],
    srcs = glob(["sub/**"]),
    cmd = "( for f in $(SRCS); do echo $$f; done ; ) > $@",
)

bin.sh:

#!/bin/bash
echo "Hello test"
find .
false

Space aren't supported in data dependencies:

  $ bazel build :bin
INFO: Analysed target //:bin (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /tmp/aaaaaaaa.aaa/runfile_spaces/BUILD:1:1: Creating runfiles tree bazel-out/k8-fastbuild/bin/bin.runfiles failed: build-runfiles failed: error executing command 
  (cd /usr/local/google/home/laszlocsomor/.cache/bazel/_bazel_laszlocsomor/b8ecdd60a1b4498304771172bdb140cf/execroot/__main__ && \
  exec env - \
    PATH=/usr/local/google/home/laszlocsomor/bazel-releases/current/bin:/usr/local/google/home/laszlocsomor/bazel-releases/current/bin:/usr/lib/google-golang/bin:/usr/local/buildtools/java/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /usr/local/google/home/laszlocsomor/.cache/bazel/_bazel_laszlocsomor/install/4cfcf40fe067e89c8f5c38e156f8d8ca/_embedded_binaries/build-runfiles bazel-out/k8-fastbuild/bin/bin.runfiles_manifest bazel-out/k8-fastbuild/bin/bin.runfiles): Process exited with status 1: Process exited with status 1
/usr/local/google/home/laszlocsomor/.cache/bazel/_bazel_laszlocsomor/install/4cfcf40fe067e89c8f5c38e156f8d8ca/_embedded_binaries/build-runfiles (args bazel-out/k8-fastbuild/bin/bin.runfiles_manifest bazel-out/k8-fastbuild/bin/bin.runfiles): link or target filename contains space on line 3: '__main__/sub/a b/bar/x.txt /tmp/aaaaaaaa.aaa/runfile_spaces/sub/a b/bar/x.txt'

Target //:bin failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.187s, Critical Path: 0.01s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

Spaces don't play well with genrule's $(SRCS).

  $ bazel build :gen && cat bazel-genfiles/gen.txt
...
INFO: Build completed successfully, 1 total action
sub/a
b/bar/x.txt

[matthewjh]

Is anyone working on a long term solution to this problem? Having this issue on MacOS with rules_webtesting.

Disclaimer: Just some hint/trick that might be useful as a workaround for some folks. This still needs to be fixed.

I just wanted to post what we have been doing in the majority of Bazel projects for rules_webtesting compatibility (where the Chromium darwin binaries contain a space). In short: The runfile space error only seems to occur when the runfile build links are being created. With the --nobuild_runfile_trees flag the runfile tree creation is deferred until actually needed (this had CI cache improvements with RBE for us as well).

The flag solves the problem nicely (for us at least) because the runfile trees are not actually created on Darwin where the sandbox strategy does not rely on runfile build links at all (more details on this here: 0324607)

# Do not build runfile forests by default. If an execution strategy relies on runfile
# forests, the forest is created on-demand. See: https://github.com/bazelbuild/bazel/issues/6627
# and https://github.com/bazelbuild/bazel/commit/03246077f948f2790a83520e7dccc2625650e6df
# Note: This also works around an issue where Bazel does not support spaces in runfiles. The
# Chromium app files for Darwin contain files with spaces and this would break. For darwin though,
# the sandbox strategy is used anyway and runfile forests are not needed.
# Related Bazel bug: https://github.com/bazelbuild/bazel/issues/4327.
build --nobuild_runfile_links

That works, until you need to debug a browser test using bazel run, in which case runfiles try to build. Or is there some way to debug in a live browser using bazel test?

[keith]

Seems like there's no workaround for this in cases where you're building coverage so your dependencies' source files are linked as runfiles and could have spaces.

EDIT: Looks like --experimental_inprocess_symlink_creation works around this, but I'm not sure if that breaks something else

[fmeum]

@lberki Do you happen to have data on the drawbacks of --experimental_inprocess_symlink_creation?

[lberki]

No data than I can take off the shelf, sorry. At Google, our runfiles trees are large enough that it stopped being feasible like a decade ago.

[meisterT]

cc @tjgq

[tjgq]

To clarify (had a chat with Lukacs internally) - at Google we use a third implementation that isn't available in Bazel.

Unfortunately --experimental_inprocess_symlink_creation won't work if you're also setting --nobuild_runfile_links. I think this was an implementation oversight - when we added the flag we forgot to also update RunfilesTreeUpdater, which creates the symlink trees just-in-time for local actions when --nobuild_runfile_links is set.

My personal opinion is that we should fix this issue [EDIT: it has been fixed in 7b87ae1] and then attempt to make in-process the default across the board (for Bazel) and delete the out-of-process implementation, but I'm not planning to work on it right now (although I might end up making incremental progress towards it while fixing #18580).

[fmeum]

@tjgq Could you briefly explain the key properties in which Google's implementation differs from the ones available in Bazel?

[tjgq]

We have an output tree backed by a FUSE daemon and directly tell the daemon to create the symlinks in the FUSE filesystem. This is similar to what's being discussed at #12823.

[mikhail-g]

Is there a way to use a delimiter other then a whitespace for Runfiles manifest entry? e.g. , : or a tab symbol

[sluongng]

It seems like the workaround/fix for this issue is to turn on --experimental_inprocess_symlink_creation.

The only drawback mentioned above was fixed in 7b87ae1 which landed in Bazel 7.0.0.

There has been another inquiry on Slack about spaces in the runfiles. I suggest we turn the flag on by default in 7.3.0.

As for the manifest file itself, I think #14676 is "almost" there but needs some additional effort to take it over the finish line. If folks are interested in making the runfiles manifest format a bit easier to parse, take a look there.