src/main/tools/process-wrapper-legacy.cc:80: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/turbine_direct_graal, ...)": No such file or directory

[vorburger]

bazel-contrib/rules_jvm_external#1451 follow-up,

but this time in the right project repo (hopefully) - if I understand correctly?

In enola-dev/enola#1730 (comment),

where I'm having (too much...) fun with running a Bazel Java build inside a reproducible Nix package "derivation" build,

when running nix run . on https://github.com/enola-dev/enola/pull/1873/files, the @dotdoom and me are running into:

ERROR: /build/home/.cache/bazel/_bazel_nixbld/c122cde4e9cdfc8e96d0ecdf2a867384/external/rules_jvm_external+/private/tools/java/com/github/bazelbuild/rules_jvm_external/BUILD:3:13: Compiling Java headers external/rules_jvm_external+/private/tools/java/com/github/bazelbuild/rules_jvm_external/librules_jvm_external-hjar.jar (3 source files) [for tool] failed: (Exit 1): turbine_direct_graal failed: error executing Turbine command (from target @@rules_jvm_external+//private/tools/java/com/github/bazelbuild/rules_jvm_external:rules_jvm_external) external/rules_java++toolchains+remote_java_tools_linux/java_tools/turbine_direct_graal '-Dturbine.ctSymPath=external/rules_java++toolchains+local_jdk/lib/ct.sym' --output ... (remaining 30 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging src/main/tools/process-wrapper-legacy.cc:80: "execvp(external/rules_java++toolchains+remote_java_tools_linux/java_tools/turbine_direct_graal, ...)": No such file or directory

Do you have any suggestions on what could be causing this, and how to fix it?

Note that this Nix package "derivation" build is a restricted sandbox, with a controlled PATH,

within which we run bazel build --vendor_dir=...

I'm struggling to fully grasp what the error message above is trying to tell us.

Surely, whatever exactly that turbine_direct_graal binary is "available"? It's presumably something from rules_java, right?

That "No such file or directory" more likely means... like, what? It's perhaps missing a dynamically linked library that it depends on? That may well be missing in that restrictive Nix sandbox environment. Wouldn't it be simpler to make that binary statically linked? (If this is even the root cause; I'm a little bit out of my depth here.)

@meteorcloudy @cushon @fmeum perhaps you have thoughts about this.

[fmeum]

This is essentially a stock GraalVM CE native image produced at https://github.com/bazelbuild/bazel/blob/370c617f4799c98d80f95e7ce2f5cefcddfda2aa/src/java_tools/buildjar/java/com/google/devtools/build/java/turbine/BUILD#L72. It statically links everything except glibc.

It may be possible to use toolchains_musl to produce a static executable along the lines of https://www.graalvm.org/latest/reference-manual/native-image/guides/build-static-executables/. Would you be interested in giving that a try?

The alternative is to fall back to the java_binary tool, but that's roughly 5x slower since it's dominated by startup time.

[vorburger]

This is essentially a stock GraalVM CE native image produced at https://github.com/bazelbuild/bazel/blob/370c617f4799c98d80f95e7ce2f5cefcddfda2aa/src/java_tools/buildjar/java/com/google/devtools/build/java/turbine/BUILD#L72. It statically links everything except glibc.

Thank You; Interesting! I would hope that even a Nix sandbox has a glibc, but perhaps as a 1st next step I'll further explore if I need to explicitly add it somehow to its build environment, or something like that.

The alternative is to fall back to the java_binary tool, but that's roughly 5x slower since it's dominated by startup time.

As I would be quite happy to go with "at least it works" over "fast", at least to start with, may I ask how would one configure rules_java to use the java_binary instead of that native 🌬️ Turbine?

It may be possible to use toolchains_musl to produce a static executable along the lines of https://www.graalvm.org/latest/reference-manual/native-image/guides/build-static-executables/. Would you be interested in giving that a try?

Perhaps, if I can not get it work any other way... 😆

[fmeum]

You can create a custom non-prebuilt Java toolchain with this config dict merged into the default config:

rules_java/toolchains/default_java_toolchain.bzl

Line 138 in 3257e83

NONPREBUILT_TOOLCHAIN_CONFIGURATION = dict(

[vorburger]

Thank You; Interesting! I would hope that even a Nix sandbox has a glibc, but perhaps as a 1st next step I'll further explore if I need to explicitly add it somehow to its build environment, or something like that.

I (briefly) tried explicitly adding pkgs.glibc to both the buildInputs and nativeBuildInputs, but it doesn't make a difference.

And glibc likely is already available anyway, how could it not be.

More likely, the fact that Nix put libraries and binaries in weird places under /nix might be causing this.

The alternative is to fall back to the java_binary tool, but that's roughly 5x slower since it's dominated by startup time.

You can create a custom non-prebuilt Java toolchain with this config dict merged into the default config:

So just NONPREBUILT_TOOLCHAIN_CONFIGURATION here and here? That seemed simple enough, so I've tried that, but now it fails with:

ERROR: /build/home/.cache/bazel/_bazel_nixbld/229eeb5b2cce4922b22dedd13ee5f872/external/rules_jvm_external++maven+maven/BUILD:12682:11: Stamping the manifest of @@rules_jvm_external++maven+maven//:org_codehaus_mojo_animal_sniffer_annotations failed: (Exit 1): AddJarManifestEntry failed: error executing StampJarManifest command (from target @@rules_jvm_external++maven+maven//:org_codehaus_mojo_animal_sniffer_annotations) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_jvm_external+/private/tools/java/com/github/bazelbuild/rules_jvm_external/jar/AddJarManifestEntry --source ... (remaining 5 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging src/main/tools/process-wrapper-legacy.cc:80: "execvp(bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_jvm_external+/private/tools/java/com/github/bazelbuild/rules_jvm_external/jar/AddJarManifestEntry, ...)": No such file or directory

which likely has a similar root cause.

Is there any easy enough way to get this StampJarManifest tool be a pure Java tool as well, instead of a dynamically linked native one?

[cushon]

I think that's from https://github.com/bazel-contrib/rules_jvm_external. I'm not actually seeing where it's using native-image or anything, it just looks like a java_binary?

• https://github.com/bazel-contrib/rules_jvm_external/blob/786201f360f227b409129ebc5a80f9b624b63ca6/private/rules/jvm_import.bzl#L41
• https://github.com/bazel-contrib/rules_jvm_external/blob/786201f360f227b409129ebc5a80f9b624b63ca6/private/tools/java/com/github/bazelbuild/rules_jvm_external/jar/BUILD#L4

[vorburger]

In a conversation with Gemini, I've learnt a little bit more than I previously knew about ld-linux, and researching and reading up more about it, figured out that as being the root cause of these problems. (Run e.g. readelf -l /usr/bin/ls| grep "program interpreter" and ldd /usr/bin/ls and man ld-linux to understand.)

So, the "correct" solution isn't working around it (as above), but instead making Nix run Bazel in a FHS (see Wikipedia and Spec) compliant environment - which Nix is not, by default. But it can be made so apparently, using buildFHSEnv (which is now the same as its predecessor buildFHSUserEnv you'll find mentioned on older blog posts).

Here are related references, for anyone else ever finding this issue again through a web search in the future:

• https://nix.dev/guides/faq.html#how-to-run-non-nix-executables
• https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments
• https://nixos.wiki/wiki/Bazel
• https://www.rugu.dev/en/blog/nixos-precompiled/
• https://www.alexghr.me/blog/til-nix-flake-fhs/
• https://discourse.nixos.org/t/flakes-way-of-creating-a-fhs-environment/20821/4

BTW @cushon do you understand why AddJarManifestEntry, being just a java_binary, fails with execvp(bazel-out/.../rules_jvm_external/jar/AddJarManifestEntry, ...)? I'm missing a bit java -jar ... AddJarManifestEntry_deploy.jar sorta line there... just for my education, do you happen to know why this is so? @meteorcloudy is "launcher" in Bazel's execvp related code "hiding" the real command, and "simplifying" it for output, when running a java_binary? That's quite confusing.

I haven't actually tried out the solution yet, but we'll do that back over in enola-dev/enola#1730.

I guess having learnt that, I can close this issue here? I was wondering if it's something worth documenting more clearly somewhere in Bazel... but it's really more of a very Nix specific issue, isn't it? So... probably moot - what do you think?

[meteorcloudy]

From the error message, it looks like it's still caused by bazelbuild/bazel#27156, are you sure you are not putting your vendor dir or any output base under /tmp?

[vorburger]

From the error message, it looks like it's still caused by bazelbuild/bazel#27156, are you sure you are not putting your vendor dir or any output base under /tmp?

FYI someone just saw this again, not running under the /tmp but on an (older) Intel (not ARM) Mac in enola-dev/enola#1915 (comment) (that's WITH enola-dev/enola#1924 WIP, wait for that PR to be merged, just in case you want to reproduce it).