runfiles: provide language-specific runfiles libraries #4460

laszlocsomor · 2018-01-15T14:31:01Z

Update(2018/04/17): link to design doc.

Blockers

Blocking:

"Windows, testing: make //src/test/shell/... run on Windows" (Windows, testing: make //src/test/shell/... run on Windows #4930)

Feature requests: what underlying problem are you trying to solve with this feature?

FR: provide language-specific runfiles libraries, with implementations of platform-independent rlocation, for the main languages: Bash, Java, C++, Python.

Motivation:

easier for users: they don't need to worry about runfiles semantics, just include a library and use runfiles the same way on every platform
easier for Bazel developers: we can change runfiles semantics if we need to, and only need to change the libraries and release Bazel for people to get the new semantics

The text was updated successfully, but these errors were encountered:

laszlocsomor · 2018-01-15T14:31:36Z

related: #3839

Cut all dependencies of the Java runfiles library, so we can include its sources in @bazel_tools, and any Java project can safely depend on it without worrying about pulling in an incompatible version of Guava for example. See #4460 Change-Id: I4a9314c56a2c9dc3afdf53669429750d8fb81e3b PiperOrigin-RevId: 181968558

Also make most targets in `//src/tools/runfiles` private. The user should depend on `@bazel_tools//tools/runfiles:$LANG-runfiles` instead. See #4460 RELNOTES[NEW]: java,runfiles: You can now depend on `@bazel_tools//tools/runfiles:java-runfiles` to get a platform-independent runfiles library for Java. See JavaDoc of https://github.com/bazelbuild/bazel/blob/master/src/tools/runfiles/java/com/google/devtools/build/runfiles/Runfiles.java for usage information. Change-Id: Iba9113453222ae74ce42a324272711f613104891 PiperOrigin-RevId: 182022851

Also update the Python stub script template to set $RUNFILES_MANIFEST_FILE or $RUNFILES_DIR so the runfiles library only needs to look for those. See #4460 RELNOTES[NEW]: python,runfiles: You can now depend on `@bazel_tools//tools/runfiles:py-runfiles` to get a platform-independent runfiles library for Python. See DocString of https://github.com/bazelbuild/bazel/blob/master/src/tools/runfiles/runfiles.py for usage information. Change-Id: I4f68a11cb59f2782e5203e39fe60cc66b46023a2 PiperOrigin-RevId: 184515490

Update the runfiles test's mock Python binary to run another mock Python binary and assert that the outer binary propagates the runfiles information to the inner one. See #4460 Change-Id: I3bba799a376642196777f7bc988837084a53bc93 PiperOrigin-RevId: 184650962

Move the mock java file in //src/test/py/bazel:runfiles_test to an actual file in the source tree. This change makes the test more readable, and the file more easily discoverable. See #4460 Change-Id: I94b033f63b218554b0e0e11814fde4c13bf7cca0 PiperOrigin-RevId: 184692391

The Python runfiles library will no longer pick up the runfiles directory from $TEST_SRCDIR. This is in accordance with the Java runfiles library, see the motivation in #4598 The commit also adds a test for this functionality as well as exercising that the runfiles library can find the runfiles directory even if no RUNFILES_* envvars are exported. See #4598 See #4460 Change-Id: I6e4d154cdca66b29d7c6e77743cf8109638238b5 PiperOrigin-RevId: 185009819

Rename the runfiles library's Runfiles.EnvVar() method to EnvVars(), indicating that it may return multiple entries in the dict. In the future this method will return multiple keys, but I want to update the interface ASAP so that it is stable after it's released in Bazel 0.12.0. See #4460 Change-Id: I8ab3a2a64736ff746dd96fea80ba2f5356dcfcc3 PiperOrigin-RevId: 185118832

Add a test to //src/test/py/bazel:runfiles_test that ensures that Java binaries using the Java runfiles library in @bazel_tools can find their runfiles, even if JAVA_RUNFILES and the RUNFILES_* envvars are undefined. There's already a shell script test for this in //src/test/shell/bazel:java_launcher_test, but we don't run that on Windows, shell tests are slower on Windows than on Linux anyway, and adding this test was very little work. Plus I'll need the new Bar.java file for other tests that are coming. See #4460 Change-Id: Ie0fa51046c3f3a1e5ece7a6206131261a5a5b1f8 PiperOrigin-RevId: 185137988

The Windows native {java,py,sh}_binary launcher now picks up the RUNFILES_MANIFEST_FILE and RUNFILES_DIR envvars from the environment. This enables running such binaries even when they are data-dependencies of other binaries and thus have no runfiles (manifest or directory) of their own. See #4460 Change-Id: I1bc8e30f81b9932ea7c96ded2717ac9d0600403d PiperOrigin-RevId: 185656004

Add "JAVA_RUNFILES" (and "RUNFILES_DIR") to the dict that Runfiles.EnvVars() returns in the Python runfiles library, so Python programs that use the Python runfiles library in @bazel_tools can now run java_binary data-dependencies as subprocesses and the latter will find the runfiles. See #4460 Change-Id: I0566f6d3c68631a1d99e67964fbe8019ee82324b PiperOrigin-RevId: 185812948

Refactor the runfiles library tests so adding the upcoming C++ runfiles library's tests will be easy. See #4460 Change-Id: Ib52a0ae1f20cc870fa1d45a15644ae1666600be2 PiperOrigin-RevId: 186268961

Implement the foundation of the C++ runfiles library, along with a directory-based Runfiles implementation. Subsequent commits will add more feataures: - manifest-based runfiles handling - creating list of envvars to pass to child processes - automatic Runfiles creation based on argv[0] and the envvars of this process See #4460 Change-Id: Id5a38619a1ae874499f04523863081559360410c PiperOrigin-RevId: 187031518

Implement a manifest-based Runfiles object, add tests and test utils (MockFile). Subsequent commits will add more feataures: - creating list of envvars to pass to child processes - automatic Runfiles creation based on argv[0] and the envvars of this process See #4460 Change-Id: I4376ede3ac00241688ff16a36ed596fb08f13a72 PiperOrigin-RevId: 187318502

Implement Runfiles::EnvVars so it's now possible to propagate runfiles to subprocesses via environment variables (RUNFILES_MANIFEST_FILE and RUNFILES_DIR). Subsequent commits will add more feataures: - automatic Runfiles creation based on argv[0] and the envvars of this process See #4460 Change-Id: If9a37b1be13b9cbacf21a496305d60444fd660b2 PiperOrigin-RevId: 187858830

The rlocation functions in the Python and Java runfiles libraries (under @bazel_tools//tools/runfiles) now consistently return the argument itself if it is an absolute path. If the argument is a driveless absolute Windows path (e.g. "\\windows\\system32") then rlocation reports an error. See bazelbuild#4460 Change-Id: I80474f7cc4736a571bf113438a916f71c36a344d

The rlocation functions in the Python and Java runfiles libraries (under @bazel_tools//tools/runfiles) now consistently return the argument itself if it is an absolute path. If the argument is a driveless absolute Windows path (e.g. "\\windows\\system32") then rlocation reports an error. See #4460 Change-Id: I80474f7cc4736a571bf113438a916f71c36a344d Closes #4806. Change-Id: I80474f7cc4736a571bf113438a916f71c36a344d PiperOrigin-RevId: 188453982

See bazelbuild#4460 Change-Id: Idf29e85c485c3dd037c3506445bbb3cb141e3968

Implement Runfiles::Create method, which inspects the RUNFILES_* envvars as well as the availability of a runfiles manifest or -directory near argv[0], and creates a Runfiles* object based on that. Also add documentation and usage examples. Subsequent commit will add integration tests. See #4460 Change-Id: I2aa433d460826999e698597205ae7663b0e84dd3 PiperOrigin-RevId: 189555342

See bazelbuild#4460 Change-Id: Idf29e85c485c3dd037c3506445bbb3cb141e3968

Add a new package that'll host the Bash runfiles library. See bazelbuild#4460 Change-Id: Id738922a1b626c09c03a226573cf002cc41cc524

See bazelbuild#4460 Change-Id: Icf38b28ad492e7b7fc431844d31dc19e74ac663e

See bazelbuild#4460 RELNOTES[NEW]: Bash, runfiles: we added a new runfiles library for Bash, see https://github.com/bazelbuild/bazel/blob/master/tools/bash/runfiles/runfiles.sh for usage information Change-Id: Icf38b28ad492e7b7fc431844d31dc19e74ac663e

Move the C++ runfiles library to the location of the rest of the C++ tools. We have not yet announced nor released the C++ runfiles library so moving it is fine. See bazelbuild#4460 Change-Id: I1732ef1eaff880cae05b7d218a3b1c0461a6b029

Compute the list of environment variables when creating the runfiles library instead of computing it on-demand. Since we already have the manifest and directory path from Runfiles::PathsFrom, creating the vector is cheap, though we have to store it even if we don't use it later. In a subsequent change I'll merge the manifest- and directory-based implementations into RunfilesImpl, and collapse RunfilesImpl into the Runfiles base class. The point of that is to make the runfiles library be able to handle both the directory-based and the manifest-based case simultaneously, dependening on what is available to it in the filesystem. See bazelbuild#4460 Change-Id: I56310423528df2d0f7494f45904150193368e2d6

The new method discovers the runfiles manifest and runfiles directory using the values of the RUNFILES_MANIFEST_FILE and RUNFILES_DIR envvars (if specified), and if needed, also looks for them next to sys.argv[0]. This commit is a copy of bazelbuild@9f2b052 ported from C++ to Python. See bazelbuild#4460 Change-Id: I6916366ca73575703fe39ce69020eec3b54457bf

Compute the list of environment variables when creating the runfiles library instead of computing it on-demand. Since we already have the manifest and directory path from Runfiles::PathsFrom, creating the vector is cheap, though we have to store it even if we don't use it later. In a subsequent change I'll merge the manifest- and directory-based implementations into RunfilesImpl, and collapse RunfilesImpl into the Runfiles base class. The point of that is to make the runfiles library be able to handle both the directory-based and the manifest-based case simultaneously, dependening on what is available to it in the filesystem. See #4460 Change-Id: I56310423528df2d0f7494f45904150193368e2d6 Closes #5218. Change-Id: I73d1f44611c4b0a73c41162319d0ba7a8a4ae6d7 PiperOrigin-RevId: 197543865

The new method discovers the runfiles manifest and runfiles directory using the values of the RUNFILES_MANIFEST_FILE and RUNFILES_DIR envvars (if specified), and if needed, also looks for them next to sys.argv[0]. This commit is a copy of 9f2b052 ported from C++ to Python. See #4460 Change-Id: I6916366ca73575703fe39ce69020eec3b54457bf Closes #5233. Change-Id: I6916366ca73575703fe39ce69020eec3b54457bf PiperOrigin-RevId: 197544480

Change the C++ runfiles library so it can use the runfiles manifest (if present) and the runfiles directory (if present) simultaneously. If the Runfiles object fails to look up a runfile from the manifest (or the manifest didn't exist or wasn't found) then it looks it up from the runfiles directory. This change allows using the same binary with and without a runfiles tree, which is often what sets local and remote runs apart. bazelbuild#4460 Change-Id: Iae879ff084ba084fcd7c111638ddeae4c6754f4f

Change the C++ runfiles library so it can use the runfiles manifest (if present) and the runfiles directory (if present) simultaneously. If the Runfiles object fails to look up a runfile from the manifest (or the manifest didn't exist or wasn't found) then it looks it up from the runfiles directory. This change allows using the same binary with and without a runfiles tree, which is often what sets local and remote runs apart. #4460 Change-Id: Iae879ff084ba084fcd7c111638ddeae4c6754f4f Closes #5235. Change-Id: Iae879ff084ba084fcd7c111638ddeae4c6754f4f PiperOrigin-RevId: 197726412

Relevant Bazel issue: bazelbuild/bazel#4460. I followed the approach documented here: https://docs.google.com/document/d/e/2PACX-1vSDIrFnFvEYhKsCMdGdD40wZRBX3m3aZ5HhVj4CtHPmiXKDCxioTUbYsDydjKtFDAzER5eg7OjJWs3V/pub Other than not bothing yet with `RUNFILES_MANIFEST`, which is only needed on WINDOWS.

- Update all runfiles libraries to have the same header comment format, including the build rule to depend on the namespace / header / module to import - Fix runfiles_test.cc to include the right files (recent refactoring commit has split src/main/cpp/util/path.cc from file_<platform>.cc) - Change exported variable _rlocation_isabs_pattern in runfiles.bash to be upper-case, so it is visibly a variable and not a function. See bazelbuild#4460 Change-Id: I17e18308506ab9f5c9f410ef6bc6b9df912d42a9

- Update all runfiles libraries to have the same header comment format, including the build rule to depend on the namespace / header / module to import - Fix runfiles_test.cc to include the right files (recent refactoring commit has split src/main/cpp/util/path.cc from file_[platform].cc) - Change exported variable _rlocation_isabs_pattern in runfiles.bash to be upper-case, so it is visibly a variable and not a function. See #4460 Change-Id: I17e18308506ab9f5c9f410ef6bc6b9df912d42a9 Closes #5481. Change-Id: I17e18308506ab9f5c9f410ef6bc6b9df912d42a9 PiperOrigin-RevId: 202291629

laszlocsomor · 2018-07-03T13:53:16Z

Done -- Bazel 0.15.0 contains the runfiles library for Python, Bash, C++, and Java, and they all support rlocation(). None of them supports rfind() yet (see #4334).

alexeagle · 2018-07-10T02:22:08Z

FYI, on my Windows machine the grep and cut dependencies in the runfiles.bash initializer block don't work. In cmd.exe and cmder.exe I get a version mismatch on a cygwin1.dll, in PowerShell grep and cut aren't found.
Does this feature require new installation instructions on Windows?

laszlocsomor · 2018-07-10T07:29:35Z

Thanks for reporting that! How do you run the binary -- directly from bazel-bin, using "bazel run", or as part of another rule (e.g. genrule)?

laszlocsomor · 2018-07-10T09:51:28Z

Meanwhile I ran into the same problem. @meteorcloudy knows how to fix it -- we just need to use bash -l in the sh_binary's launcher.

meteorcloudy · 2018-07-11T12:27:47Z

After discussion we decided not to use login shell (-l), because it will not only override the original PATH env var, but also do other things we don't need (eg. changing current directory, run shell configuration files)
Instead we will add the bash bin directory to PATH to make the bin tools available.

alexeagle · 2018-07-12T00:48:18Z

Would you like a separate issue to track that? Or maybe re-open this one?

…

On Wed, Jul 11, 2018 at 5:28 AM Yun Peng ***@***.***> wrote: After discussion we decided not to use login shell (-l), because it will not only override the original PATH env var, but also do other things we don't need (eg. changing current directory, run shell configuration files) Instead we will add the bash bin directory to PATH to make the bin tools available. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4460 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC5I5kDvGa7u1uFuuTR459TjoGxzz_Oks5uFe9VgaJpZM4RefpK> .

alexeagle · 2018-07-31T22:33:45Z

Opened a separate issue since this is still broken in Bazel 0.16

- Update all runfiles libraries to have the same header comment format, including the build rule to depend on the namespace / header / module to import - Fix runfiles_test.cc to include the right files (recent refactoring commit has split src/main/cpp/util/path.cc from file_[platform].cc) - Change exported variable _rlocation_isabs_pattern in runfiles.bash to be upper-case, so it is visibly a variable and not a function. See bazelbuild#4460 Change-Id: I17e18308506ab9f5c9f410ef6bc6b9df912d42a9 Closes bazelbuild#5481. Change-Id: I17e18308506ab9f5c9f410ef6bc6b9df912d42a9 PiperOrigin-RevId: 202291629

laszlocsomor self-assigned this Jan 15, 2018

laszlocsomor added type: feature request P1 I'll work on this now. (Assignee required) windows-q1-2018 labels Jan 15, 2018

laszlocsomor mentioned this issue Feb 13, 2018

external/examples/hello_runfiles/hello_runfiles_test failing in Bazel release 0.11.0rc3 bazelbuild/rules_rust#68

Closed

laszlocsomor mentioned this issue Mar 8, 2018

runfiles,py,java: rlocation accepts absolute path #4806

Closed

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Mar 16, 2018

Bash,runfiles: add a runfiles library

8cab2f5

See bazelbuild#4460 Change-Id: Idf29e85c485c3dd037c3506445bbb3cb141e3968

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Mar 19, 2018

Bash,runfiles: add a runfiles library

a782a67

See bazelbuild#4460 Change-Id: Idf29e85c485c3dd037c3506445bbb3cb141e3968

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Mar 19, 2018

runfiles,Bash: add a package for runfiles library

cd8eb70

Add a new package that'll host the Bash runfiles library. See bazelbuild#4460 Change-Id: Id738922a1b626c09c03a226573cf002cc41cc524

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Mar 19, 2018

Bash,runfiles: add a runfiles library

69c9c09

See bazelbuild#4460 Change-Id: Icf38b28ad492e7b7fc431844d31dc19e74ac663e

laszlocsomor mentioned this issue May 18, 2018

cpp runfiles library: precompute envvars map #5218

Closed

laszlocsomor mentioned this issue May 22, 2018

runfiles,python: add a method to discover runfiles #5233

Closed

laszlocsomor mentioned this issue May 22, 2018

runfiles,cc: Runfiles uses manifest AND directory #5235

Closed

evanj mentioned this issue Jun 1, 2018

RUNFILES detection in executable scripts is hard #4054

Open

judah mentioned this issue Jun 8, 2018

Add a library for looking up runfiles. tweag/rules_haskell#302

Merged

laszlocsomor mentioned this issue Jun 27, 2018

runfiles libraries: fix tests and comments #5481

Closed

laszlocsomor closed this as completed Jul 3, 2018

alexeagle mentioned this issue Jul 31, 2018

runfiles.bash helper broken on Windows #5724

Closed

pauldraper mentioned this issue Oct 14, 2019

Document runfiles #10022

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runfiles: provide language-specific runfiles libraries #4460

runfiles: provide language-specific runfiles libraries #4460

laszlocsomor commented Jan 15, 2018 •

edited

Loading

laszlocsomor commented Jan 15, 2018

laszlocsomor commented Jul 3, 2018

alexeagle commented Jul 10, 2018

laszlocsomor commented Jul 10, 2018

laszlocsomor commented Jul 10, 2018

meteorcloudy commented Jul 11, 2018

alexeagle commented Jul 12, 2018 via email

alexeagle commented Jul 31, 2018

runfiles: provide language-specific runfiles libraries #4460

runfiles: provide language-specific runfiles libraries #4460

Comments

laszlocsomor commented Jan 15, 2018 • edited Loading

Blockers

Feature requests: what underlying problem are you trying to solve with this feature?

laszlocsomor commented Jan 15, 2018

laszlocsomor commented Jul 3, 2018

alexeagle commented Jul 10, 2018

laszlocsomor commented Jul 10, 2018

laszlocsomor commented Jul 10, 2018

meteorcloudy commented Jul 11, 2018

alexeagle commented Jul 12, 2018 via email

alexeagle commented Jul 31, 2018

laszlocsomor commented Jan 15, 2018 •

edited

Loading