After BuildPack 8.7.0, JVM Launch Error in JRE 8

[bungabear]

What happened?

• What were you attempting to do?
  Run a docker image built by bootBuildGradle with Gradle 7.1, Java 8 and Spring Boot 2.5.4.

• What did you expect to happen?
  Successfully run a docker image

• What was the actual behavior? Please provide log output, if possible.
  When I run a docker image built by BuildPack, Some JVM error occurs. This error appears after 8.7.0 released.
  below is a log header and full log file hs_err_pid1.log

  #
  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  SIGSEGV (0xb) at pc=0x00007f2fc8dc22f7, pid=1, tid=0x00007f2fc9f2a700
  #
  # JRE version:  (8.0_302-b08) (build )
  # Java VM: OpenJDK 64-Bit Server VM (25.302-b08 mixed mode linux-amd64 compressed oops)
  # Problematic frame:
  # V  [libjvm.so+0xb2a2f7]  VirtualMemoryTracker::add_reserved_region(unsigned char*, unsigned long, NativeCallStack const&, MemoryType, bool)+0x77
  #
  # Core dump written. Default location: /workspace/core or core.1
  #
  # If you would like to submit a bug report, please visit:
  #   https://bell-sw.com/support
  #
  

Build Configuration

• What platform (pack, kpack, tekton buildpacks plugin, etc.) are you
  using? Please include a version.
  I have no idea. I used only pure bootBuildImage

• What buildpacks are you using? Please include versions.
  A log of bootBuildImage with JVM errored verion,
  error_version_log.txt

  A log of bootBuildImage at yesterday that successfully launched. and it is same when force downgrade to 8.6.0
  success_version_log.txt

  I think JVM Option changes after BuildPack 8.7.0 cause error.
  I checked UnlockDiagnosticVMOptions NativeMemoryTracking PrintNMTStatistics options are added
  And I've tried set docker env or gradle build option with JAVA_TOOL_OPTIONS to disable those things. But not works.

  So, I forced downgrade builderpack version to 8.6.0 and it works.

  bootBuildImage {
    builder = "paketobuildpacks/builder@sha256:3c5f88e55da475957bd03b4d6e852e0938bb73ee1a5184ae291a7566ca83d97f"
    runImage = "paketobuildpacks/run@sha256:562ddd10160a2629168a0108894766b46dff7f4b0db263ecabe35cd56fcbe56d"
  }
  

• What builder are you using? If custom, can you provide the output from pack inspect-builder <builder>?
  No Customs

• Can you provide a sample app or relevant configuration (buildpack.yml,
  nginx.conf, etc.)?
  Simple docker image build with run ./gradlew bootBuildImage with Spring Boot project in Github Action.
  And Simple launch with Docker docker run [built-image]

  name: ci
  
  on:
    push:
      branches:
        - 'release'
  
  jobs:
    docker:
      runs-on: ubuntu-latest
      steps:
        - name: Set up Docker Buildx
          uses: docker/setup-buildx-action@v1
        - uses: actions/checkout@v2
        - name: Set up JDK 1.8
          uses: actions/setup-java@v1
          with:
            java-version: 1.8
        - name: Grant execute permission for gradlew
          run: chmod +x gradlew
        - name: Build bootBuildImage with version
          run: ./gradlew bootBuildImage

Checklist

• I have included log output.
• The log output includes an error message.
• I have included steps for reproduction.

[dmikusa]

Interesting. I see this happening with Gradle builds, but not Maven builds. I need to investigate further and see what is different here.

Good news is I can replicate this. I'll keep you posted.

[dmikusa]

OK, so not related to Gradle. It is just purely Java 8 vs 11. My sample gradle app was using Java 8 by default, so it was impacted in my initial tests. We can look into this some more. The JVM has some bugs around enabling Java NMT using JAVA_TOOL_OPTIONS, which is what's triggering this. I thought we had worked around this in our tests, but seems like it's still triggering the bug.

You can work around this for now by adding BPL_JAVA_NMT_ENABLED=false when you run your application image. This will disable Java NMT. You must do this at runtime, not build time. So if you are running your container images with Docker, then you'd use docker run -it -p 8080:8080 -e BPL_JAVA_NMT_ENABLED=false apps/maven.

[bungabear]

OK, so not related to Gradle. It is just purely Java 8 vs 11. My sample gradle app was using Java 8 by default, so it was impacted in my initial tests. We can look into this some more. The JVM has some bugs around enabling Java NMT using JAVA_TOOL_OPTIONS, which is what's triggering this. I thought we had worked around this in our tests, but seems like it's still triggering the bug.

You can work around this for now by adding BPL_JAVA_NMT_ENABLED=false when you run your application image. This will disable Java NMT. You must do this at runtime, not build time. So if you are running your container images with Docker, then you'd use docker run -it -p 8080:8080 -e BPL_JAVA_NMT_ENABLED=false apps/maven.

-e BPL_JAVA_NMT_ENABLED=false It works well. Thank you for a first aid guide.

[leeyuanzhang]

OK, so not related to Gradle. It is just purely Java 8 vs 11. My sample gradle app was using Java 8 by default, so it was impacted in my initial tests. We can look into this some more. The JVM has some bugs around enabling Java NMT using JAVA_TOOL_OPTIONS, which is what's triggering this. I thought we had worked around this in our tests, but seems like it's still triggering the bug.

You can work around this for now by adding BPL_JAVA_NMT_ENABLED=false when you run your application image. This will disable Java NMT. You must do this at runtime, not build time. So if you are running your container images with Docker, then you'd use docker run -it -p 8080:8080 -e BPL_JAVA_NMT_ENABLED=false apps/maven.

Thanks！！！

[dmikusa]

We cut a release to workaround this issue today: https://github.com/paketo-buildpacks/java/releases/tag/v5.17.0. It disables Java NMT if you're on Java 8. If you try to enable Java NMT, it's going to fail with the same error.

There will be a fix in the next Bellsoft Liberica Java 8 release so we can enable Java NMT. It should go upstream to other OpenJDK releases after, but it's unclear how long that will take.