Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot build any more after update of rules_java to 7.3.0 #20298

Closed
davido opened this issue Nov 23, 2023 · 6 comments
Closed

Cannot build any more after update of rules_java to 7.3.0 #20298

davido opened this issue Nov 23, 2023 · 6 comments
Labels

Comments

@davido
Copy link
Contributor

davido commented Nov 23, 2023

Description of the bug:

Trying to build Bazel@HEAD (6d0716d) after #20216 is failing with:

$> bazel build -s src:bazel-bin-dev
[...]
SUBCOMMAND: # @rules_jvm_external~5.2//private/tools/java/com/github/bazelbuild/rules_jvm_external/zip:zip [action 'Compiling Java headers external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/libzip-hjar.jar (1 source file) [for tool]', configuration: 068f53a2413fd0ad955d2d18c7d44f82f04beb3900c10e221b299e686a717fe8, execution platform: //:default_host_platform]
(cd /home/davido/.cache/bazel/_bazel_davido/0fa756dec521553dbe2dde6b6eac99b4/execroot/_main && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
    PATH=/home/davido/.cache/bazelisk/downloads/bazelbuild/bazel-6.4.0-linux-x86_64/bin:/home/davido/.yarn/bin:/home/davido/.config/yarn/global/node_modules/.bin:/home/davido/pgm/apache-maven-3.8.6/bin:/home/davido/bin:/home/davido/.yarn/bin:/home/davido/.config/yarn/global/node_modules/.bin:/home/davido/pgm/google-cloud-sdk/bin:/home/davido/pgm/apache-maven-3.8.6/bin:/home/davido/bin:/home/davido/bin:/usr/local/bin:/usr/bin:/bin \
  external/rules_java~7.3.0~toolchains~remote_java_tools_linux/java_tools/turbine_direct_graal --output bazel-out/k8-opt-exec-EDC14992/bin/external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/libzip-hjar.jar --output_deps bazel-out/k8-opt-exec-EDC14992/bin/external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/libzip-hjar.jdeps --bootclasspath bazel-out/k8-opt-exec-EDC14992/bin/external/rules_java~7.3.0/toolchains/platformclasspath.jar --sources external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/StableZipEntry.java --javacopts -source 11 -target 11 '-XDskipDuplicateBridges=true' '-XDcompilePolicy=simple' -g -parameters -Xep:ReturnValueIgnored:OFF -Xep:IgnoredPureGetter:OFF -Xep:EmptyTopLevelDeclaration:OFF -Xep:LenientFormatStringValidation:OFF -Xep:ReturnMissingNullable:OFF -Xep:UseCorrectAssertInTests:OFF -- --target_label @@rules_jvm_external~5.2//private/tools/java/com/github/bazelbuild/rules_jvm_external/zip:zip --reduce_classpath_mode NONE)
# Configuration: 068f53a2413fd0ad955d2d18c7d44f82f04beb3900c10e221b299e686a717fe8
# Execution platform: //:default_host_platform
ERROR: /home/davido/.cache/bazel/_bazel_davido/0fa756dec521553dbe2dde6b6eac99b4/external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/BUILD:1:13: Compiling Java headers external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/zip/libzip-hjar.jar (1 source file) [for tool] failed: (Exit 1): turbine_direct_graal failed: error executing command (from target @rules_jvm_external~5.2//private/tools/java/com/github/bazelbuild/rules_jvm_external/zip:zip) external/rules_java~7.3.0~toolchains~remote_java_tools_linux/java_tools/turbine_direct_graal --output ... (remaining 27 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
The current machine does not support all of the following CPU features that are required by the image: [CX8, CMOV, FXSR, MMX, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT, LZCNT, AVX, AVX2, BMI1, BMI2, FMA].
Please rebuild the executable with an appropriate setting of the -march option.Target //src:bazel-bin-dev failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/davido/.cache/bazel/_bazel_davido/0fa756dec521553dbe2dde6b6eac99b4/external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/jar/BUILD:3:12 Building external/rules_jvm_external~5.2/private/tools/java/com/github/bazelbuild/rules_jvm_external/jar/AddJarManifestEntry.jar (1 source file) [for tool] failed: (Exit 1): turbine_direct_graal failed: error executing command (from target @rules_jvm_external~5.2//private/tools/java/com/github/bazelbuild/rules_jvm_external/zip:zip) external/rules_java~7.3.0~toolchains~remote_java_tools_linux/java_tools/turbine_direct_graal --output ... (remaining 27 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 133.701s, Critical Path: 81.07s
INFO: 625 processes: 170 internal, 455 linux-sandbox.
FAILED: Build did NOT complete successfully

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Have a 7 year old Lenovo Thinkpad Laptop with latest Linux version installed:

$ uname -a
Linux localhost 6.5.9-1-default #1 SMP PREEMPT_DYNAMIC Wed Oct 25 10:31:37 UTC 2023 (29edc7c) x86_64 x86_64 x86_64 GNU/Linux

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

6.4.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

#20216

Have you found anything relevant by searching the web?

As pointed out in this SO thread, detecting my cpu support for x86-64-v2:

$ g++ multiversioning.c 

$ ./a.out 
x86-64-v2

Any other information, logs, or outputs that you want to share?

According to the Graal VM native image build documentation, -march option defaults to x86-64-v3 on AMD64.

Workaround is to revert the recent rule_javaupgrade:

diff --git a/MODULE.bazel b/MODULE.bazel
index e925eed382..d1e114d682 100644
--- a/MODULE.bazel
+++ b/MODULE.bazel
@@ -23,7 +23,7 @@ bazel_dep(name = "zstd-jni", version = "1.5.2-3.bcr.1")
 bazel_dep(name = "blake3", version = "1.3.3.bcr.1")
 bazel_dep(name = "zlib", version = "1.3")
 bazel_dep(name = "rules_cc", version = "0.0.9")
-bazel_dep(name = "rules_java", version = "7.3.0")
+bazel_dep(name = "rules_java", version = "7.2.0")
 bazel_dep(name = "rules_graalvm", version = "0.10.3")
 bazel_dep(name = "rules_proto", version = "5.3.0-21.7")
 bazel_dep(name = "rules_jvm_external", version = "5.2")
@fmeum
Copy link
Collaborator

fmeum commented Nov 23, 2023

I will adjust the turbine build to not rely on the more modern instruction sets.

@meteorcloudy Is there any particular level of hardware support you aim for with Bazel releases that I could follow here?

@meteorcloudy
Copy link
Member

I will adjust the turbine build to not rely on the more modern instruction sets.

Will that negatively impact the performance?

Is there any particular level of hardware support you aim for with Bazel releases that I could follow here?

Unfortunately not. Can we offer a way to switch back to the old java_binary for such cases?

@fmeum
Copy link
Collaborator

fmeum commented Nov 23, 2023

I went with #20306, downgrading the use of advanced instruction sets to what is generally considered "compatible" (e.g. matches the Linux kernel).

While there could be a slight impact on performance, the overall win of using a native image is still substantial on all devices. Once we can get this to stick in a Bazel release, we could think about shipping more optimized variants.

@sgammon
Copy link

sgammon commented Nov 26, 2023

@fmeum you may find the "-H:RuntimeCheckedCPUFeatures flag useful here at a later time in the GraalVM build, which will check at runtime and use different code paths based on instruction set availability

example of use:

-H:RuntimeCheckedCPUFeatures=AVX2,SSE,...

@davido
Copy link
Contributor Author

davido commented Nov 26, 2023

@sgammon

Thanks for the pointer. It doesn't seem to work, though.

I downloaded the latest available graalvm-jdk-17_linux-x64_bin.tar.gz, and tried to reproduce the problem.

HelloWorld.java

public class HelloWorld {
  public static void main(String[] args) {
    System.out.println("Hello, World!");
  }
}

Then I'm creating the image on machine, that supports -march=x86-64-v3, and passing: -H:RuntimeCheckedCPUFeatures=CX8,CMOV,FXSR,MMX,SSE,SSE2,SSE3,SSSE3,SSE4_1,SSE4_2,POPCNT,LZCNT,AVX,AVX2,BMI1,BMI2,FMA

davido@wizball:~/pgm/graalvm-jdk-17.0.9+11.1$ ./bin/native-image -H:RuntimeCheckedCPUFeatures=CX8,CMOV,FXSR,MMX,SSE,SSE2,SSE3,SSSE3,SSE4_1,SSE4_2,POPCNT,LZCNT,AVX,AVX2,BMI1,BMI2,FMA HelloWorld 
========================================================================================================================
GraalVM Native Image: Generating 'helloworld' (executable)...
========================================================================================================================
[1/8] Initializing...                                                                                    (6.1s @ 0.17GB)
 Java version: 17.0.9+11-LTS, vendor version: Oracle GraalVM 17.0.9+11.1
 Graal compiler: optimization level: 2, target machine: x86-64-v3, PGO: ML-inferred
 C compiler: gcc (linux, x86_64, 9.4.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
[2/8] Performing analysis...  [****]                                                                     (7.7s @ 0.24GB)
   1,864 (59.27%) of  3,145 types reachable
   1,737 (46.34%) of  3,748 fields reachable
   7,733 (35.66%) of 21,687 methods reachable
     649 types,     0 fields, and   292 methods registered for reflection
      49 types,    32 fields, and    48 methods registered for JNI access
       4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                               (1.4s @ 0.27GB)
[4/8] Parsing methods...      [**]                                                                       (2.3s @ 0.28GB)
[5/8] Inlining methods...     [***]                                                                      (0.8s @ 0.32GB)
[6/8] Compiling methods...    [[6/8] Compiling methods...    [****]                                                                    (20.9s @ 0.42GB)
[7/8] Layouting methods...    [*]                                                                        (1.1s @ 0.44GB)
[8/8] Creating image...       [**]                                                                       (3.0s @ 0.24GB)
   2.75MB (42.05%) for code area:     3,494 compilation units
   3.46MB (52.94%) for image heap:   49,012 objects and 1 resources
 335.82kB ( 5.01%) for other data
   6.54MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
   1.44MB java.base                                          548.25kB byte[] for code metadata
   1.13MB svm.jar (Native Image)                             416.92kB byte[] for java.lang.String
  69.54kB com.oracle.svm.svm_enterprise                      326.55kB java.lang.String
  33.89kB org.graalvm.nativeimage.base                       306.38kB java.lang.Class
  30.23kB org.graalvm.sdk                                    254.17kB byte[] for general heap data
  18.95kB jdk.internal.vm.ci                                 147.28kB java.util.HashMap$Node
  14.10kB jdk.internal.vm.compiler                           111.71kB char[]
   1.17kB jdk.proxy3                                          79.05kB java.lang.Object[]
   1.15kB jdk.proxy1                                          72.81kB com.oracle.svm.core.hub.DynamicHubCompanion
  360.00B jdk.proxy2                                          71.03kB byte[] for reflection metadata
   54.00B for 1 more packages                                441.61kB for 506 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 G1GC: Use the G1 GC ('--gc=G1') for improved latency and throughput.
 PGO:  Use Profile-Guided Optimizations ('--pgo') for improved throughput.
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
 QBM:  Use the quick build mode ('-Ob') to speed up builds during development.
------------------------------------------------------------------------------------------------------------------------
                        1.6s (3.5% of total time) in 234 GCs | Peak RSS: 0.92GB | CPU load: 5.85
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /home/dos/pgm/graalvm-jdk-17.0.9+11.1/helloworld (executable)
========================================================================================================================
Finished generating 'helloworld' in 44.3s.

Then I copy this image to another machine, that only supports -march=x86-64-v2 options, but getting this breakage when trying to execute it:

davido@localhost:~/pgm/graalvm-jdk-17.0.9+11.1$ ./helloworld
The current machine does not support all of the following CPU features that are required by the image: [CX8, CMOV, FXSR, MMX, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT, LZCNT, AVX, AVX2, BMI1, BMI2, FMA].

Also note, then when I'm building the same HelloWorld image on outdated machine, I'm getting this warning:

davido@localhost:~/pgm/graalvm-jdk-17.0.9+11.1$ ./bin/native-image HelloWorld
========================================================================================================================
GraalVM Native Image: Generating 'helloworld' (executable)...
========================================================================================================================
Warning: The host machine does not support all features of 'x86-64-v3'. Falling back to '-march=compatibility' for best compatibility.
[...]

@sgowroji sgowroji added the team-Rules-Java Issues for Java rules label Nov 27, 2023
copybara-service bot pushed a commit that referenced this issue Nov 27, 2023
Work towards #20298

Closes #20306.

PiperOrigin-RevId: 585558369
Change-Id: I7a1230423479c878f87e37b907078ddd05c3970e
@davido
Copy link
Contributor Author

davido commented Nov 27, 2023

This is fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants