RUNFILES detection in executable scripts is hard

[tvolkert]

In order for executable rules to be able to properly refer to files that are included as data deps, those executables need to know what the RUNFILES directory is.

There are many contexts from which an executable can be invoked, so it's common for the templates of such executable rules to contain boilerplate code that tries to deduce RUNFILES with code that looks similar to the following:

if [[ -n "$TEST_SRCDIR" && -d "$TEST_SRCDIR" ]]; then
  # use $TEST_SRCDIR if set.
  export RUNFILES="$TEST_SRCDIR"
elif [[ -z "$RUNFILES" ]]; then
  # canonicalize the entrypoint.
  pushd "$(dirname "$0")" > /dev/null
  abs_entrypoint="$(pwd -P)/$(basename "$0")"
  popd > /dev/null
  if [[ -e "${abs_entrypoint}.runfiles" ]]; then
    # runfiles dir found alongside entrypoint.
    export RUNFILES="${abs_entrypoint}.runfiles"
  elif [[ "$abs_entrypoint" == *".runfiles/"* ]]; then
    # runfiles dir found in entrypoint path.
    export RUNFILES="${abs_entrypoint%.runfiles/*}.runfiles"
  else
    # runfiles dir not found: fall back on current directory.
    export RUNFILES="$PWD"
  fi
fi

Blocks of code like this end up sprinkled throughout a project's Bazel codebase, which is (a) not very maintainable, and (b) fragile. And note that the code above only attempts to work on Mac and Linux - with Windows support, it'd look even worse.

Is there any way Bazel could guarantee that $RUNFILES will always be set when executable targets are invoked from the following contexts:

• bazel run
• bazel test
• Being invoked via a genrule

If those guarantees were made, then developers would only have to worry about deducing RUNFILES when the script was invoked directly at the command-line, in which case RUNFILES is basically <path-to-current-script>.runfiles/

[benjaminp]

I think there should be a documented algorithm for resolving an executable's runfiles tree and reference implementations of that algorithm for all builtin Bazel rules. Finding an executable's runfiles turns out be a fairly complex process with correctness implications (e.g., #3919). There are many different environments where the correct runfiles tree must be found. Unsurprisingly, no two current implementations of runfiles tree resolution work the same, so the area is ripe for standardization.

[evanj]

For people who find this in the future: It seems like finding the first part of argv[0] that ends in .runfiles may be a "good" way to do this at the moment (from comments here: bazelbuild/rules_docker#204 (comment) ). In general, this is well documented for tests (the TEST_SRCDIR environment variable will be set). This is trickier for binaries. Its even harder for scripting languages like Python, where the initial "executable" may just be a script that actually executes some other file, probably located in the runfiles tree itself. Maybe it should be the responsibility for each *_binary rule to define some interface?

[evanj]

After some additional investigation, issue #4460 is related

[jin]

cc @philwo please triage if this still relevant.

[aryeh-looker]

This is kind of a big deal, as it prevents easily using a java binary as a tool in a genrule if the java binary uses runfiles.

Caused by: java.io.IOException: Cannot find runfiles: $RUNFILES_DIR and $JAVA_RUNFILES are both unset or empty

This is with bazel 6.0.0 and Runfiles.preload().unmapped().

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.