Java::JavaLang::NoClassDefFoundError (java/sql/Date) #6608

fzakaria · 2021-03-11T01:58:22Z

Environment Information

Provide at least:

JRuby version: 9.2.13.0 & 9.2.16.0

Picked up JAVA_TOOL_OPTIONS: -Djava.library.path=/nix/store/9l06v7fc38c1x3r2iydl15ksgz0ysb82-glibc-2.32/lib
jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.9+0-adhoc..source on 11.0.9+0-adhoc..source +jit [linux-x86_64]

Operating system and platform: Linux fmzakari-glaptop 5.7.17-1rodete5-amd64 #1 SMP Debian 5.7.17-1rodete5 (2021-01-08) x86_64 GNU/Linux

Expected Behavior

Attempting to call to_java on a RubyTime object and have it return a java.sql.Date object returns NoClassDefFoundError.

What is perculiar is that I can clearly create a java.sql.Date, so it seems to be accessible via my classpath.

Here is a minimal reproduction with irb:

# let us prove first that java.sql.Date works!
irb(main):013:0> date = java.sql.Date.new(0)
=> #<Java::JavaSql::Date:0x2715644a>

# now fetch the class
irb(main):014:0> clazz = java.sql.Date.java_class
=> class java.sql.Date

# lets create a RubyTime object now
irb(main):015:0> t = Time.new(0)
=> 0000-01-01 00:00:00 -0752

# lets try to cast it to java.sql.Date
irb(main):016:0> t.to_java(clazz)
Traceback (most recent call last):
       16: from org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)
       15: from org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)
       14: from org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84)
       13: from org.jruby.RubyKernel$INVOKER$s$0$3$eval.call(RubyKernel$INVOKER$s$0$3$eval.gen)
       12: from org.jruby.RubyKernel.eval(RubyKernel.java:1048)
       11: from org.jruby.RubyKernel.evalCommon(RubyKernel.java:1086)
       10: from org.jruby.ir.interpreter.Interpreter.evalWithBinding(Interpreter.java:182)
        9: from org.jruby.ir.interpreter.Interpreter.evalCommon(Interpreter.java:158)
        8: from org.jruby.ir.interpreter.Interpreter.INTERPRET_EVAL(Interpreter.java:106)
        7: from org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
        6: from org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        5: from org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:174)
        4: from org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:375)
        3: from org.jruby.java.addons.KernelJavaAddons$INVOKER$s$to_java.call(KernelJavaAddons$INVOKER$s$to_java.gen)
        2: from org.jruby.java.addons.KernelJavaAddons.to_java(KernelJavaAddons.java:29)
        1: from org.jruby.RubyTime.toJava(RubyTime.java:1472)
Java::JavaLang::NoClassDefFoundError (java/sql/Date)

A bit of a headscratcher for me since right before the call to to_java, you can see it working.

Actual Behavior

I expect the cast to work.
If I change the JDK we are using to JDK8, I see that the results work as intended.

irb(main):001:0> clazz = java.sql.Date.java_class
=> class java.sql.Date
irb(main):002:0>  t = Time.new(0)
=> 0000-01-01 00:00:00 -0752
irb(main):003:0> t.to_java(clazz)
=> #<Java::JavaSql::Date:0x5223e5ee>

There is something funky here going on with module-loading.

The text was updated successfully, but these errors were encountered:

headius · 2021-03-11T16:13:57Z

Aha... I suspect this is due to us moving more into modularity, and this class references things outside the jruby.base module.

I think this can be worked around by adding a --add-module java.sql line to the Java options, but we should probably put that into either the startup flags or request that module when booting up RubyTime.

fzakaria · 2021-03-11T16:21:37Z

Why does it work on irb or through Ruby directly....

headius · 2021-03-11T18:48:47Z

@fzakaria That is a great question. I will try to reproduce today.

Meanwhile you could see if JAVA_OPTS="--add-module java.sql" fixes it.

FWIW in the branch where I fully modularize JRuby, I add this to our explicit requires so it should not be a problem there (#6598).

fzakaria · 2021-03-11T18:53:39Z

On JDK11 looks like --add-module was removed.
https://stackoverflow.com/questions/53993029/add-modules-flag-replacement-for-java11

❯ JAVA_OPTS="--add-module java.sql" irb
Picked up JAVA_TOOL_OPTIONS: -Djava.library.path=/nix/store/9l06v7fc38c1x3r2iydl15ksgz0ysb82-glibc-2.32/lib
Unrecognized option: --add-module
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

headius · 2021-03-11T19:46:40Z

@fzakaria Sorry, it is --add-modules 🙄

headius · 2021-03-11T20:38:38Z

Hmm I can't reproduce from command line nor irb:

[] ~/projects/jruby-9.2 $ java -version
openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.4+11)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.4+11, mixed mode)

[] ~/projects/jruby-9.2 $ jruby -e "p Time.now.to_java(java.sql.Date.java_class)"
#<Java::JavaSql::Date:0x57e4b86c>

[] ~/projects/jruby-9.2 $ jruby -e "p Time.now.to_java(java.sql.Date)"
#<Java::JavaSql::Date:0x57e4b86c>

[] ~/projects/jruby-9.2 $ jruby -X-C -e "p Time.now.to_java(java.sql.Date)"
#<Java::JavaSql::Date:0x1a717d79>

[] ~/projects/jruby-9.2 $ jruby -X-C -e "p Time.now.to_java(java.sql.Date.java_class)"
#<Java::JavaSql::Date:0x1a717d79>

[] ~/projects/jruby-9.2 $ irb
irb(main):001:0> date = java.sql.Date.new(0)
=> #<Java::JavaSql::Date:0x121ddca4>
irb(main):002:0> Time.now.to_java(java.sql.Date.java_class)
=> #<Java::JavaSql::Date:0x5fde1d64>
irb(main):003:0>

headius · 2021-03-11T20:41:24Z

(FWIW you don't need .java_class here and that API is kinda-sorta deprecated.)

fzakaria · 2021-03-12T00:04:54Z

@headius okay! I got a lot of good information for you. I opened a fix.

The jruby script does the following:

# Detect Java 9+ by the presence of a jmods directory in JAVA_HOME
if [ -d "$JAVA_HOME"/jmods ]; then
  is_java9=1
fi

add_log "  JAVACMD: $JAVACMD"
add_log "  JAVA_HOME: $JAVA_HOME"

if [ "$is_java9" ]; then
  add_log
  add_log "Detected Java modules at $JAVA_HOME/jmods"
fi

On my non-Nix Java installation this makes sense.

❯ ls -l /usr/lib/jvm/java-11-openjdk-amd64/jmods | head -n 3

.rw-r--r-- 162M root 20 Jan  1:42 java.base.jmod
.rw-r--r-- 119k root 20 Jan  1:42 java.compiler.jmod
.rw-r--r--  58k root 20 Jan  1:42 java.datatransfer.jmod

Nix however places jmods in a different directory.

❯ ls -l /nix/store/s1n0y12md0b1mvsg4ly57d43xxwbjdzq-openjdk-11.0.9+11/lib/openjdk/jmods | head -n 3
.r--r--r-- 256M fmzakari 31 Dec  1969 java.base.jmod
.r--r--r-- 119k fmzakari 31 Dec  1969 java.compiler.jmod
.r--r--r--  58k fmzakari 31 Dec  1969 java.datatransfer.jmod

Digging into how the JDK is setup (JDK11 no longer has a JRE) https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/openjdk/jre.nix#L16 I see the following:

jlink --module-path ${jdk}/lib/openjdk/jmods --add-modules ${lib.concatStringsSep "," modules} --output $out
patchelf --shrink-rpath $out/bin/* $out/lib/jexec $out/lib/jspawnhelper $out/lib/*.so $out/lib/*/*.so

I found the bug in NixPkgs and opened up NixOS/nixpkgs#116009

I wonder if there's a better approach for the JRuby script to find the module path that is more resilient.

This is investigation I've done in support of jruby/jruby#6608 where I noticed some funky issues with JRuby and module loading. Looks like that JRuby expects JAVA_HOME to have a directory called `jmod`, which is consistent with the Java Module system. Unfortunately, the top level directory for the `jre` or `jdk` /nix/store entry is not a valid JAVA_HOME since it is missing that directory. Instead it's set within `lib/openjdk`, and there is a passthru variable set accordingly. This fixes JRuby and follows many other derivations. A simple search in the code-base shows that there are many other packages that suffer this same bug.

fzakaria · 2021-03-12T00:19:58Z

Still super unclear why in irb it can resolve it without the use of to_java.
There's some magic module finding/loading going on?

fzakaria · 2021-03-12T00:23:37Z

NixOS/nixpkgs#116009 (comment) has a good description showing bug and then fixed.

headius · 2021-03-12T00:33:40Z

@fzakaria Aha! Yep, that makes perfect sense. At the very least, we are not loading properly as a module which means none of our module flags will be used and certain functionality we need to open up will not be available. I knew this would be a fragile way to detect the presence of modules but it worked okay up until now.

We do need to find a better way to detect that we will be running on a module-enabled jvm, but I do not know the answer right now. There may be a better file we can look at, or a fast way to check the version via some command. Let's do some research and see what we can figure out.

fzakaria · 2021-03-12T02:14:54Z

@headius here is what I came up with.

❯ JAVA_HOME=~/not-correct java -XshowSettings:properties -version 2>&1 | grep java.home

    java.home = /nix/store/2xv65447vkq09bpx1mfci1cx2vdvp153-openjdk-headless-11.0.9+11/lib/openjdk

Don't know if there's a slicker way to get jmod specifically.

Detecting module support based on the existence of the jmods directory was fragile for at least two reasons: * The jmods directory is only present in some builds of the JDK (not JRE) that can be used to jlink new runtime environments. * Some distributions of the JDK may omit or relocate this directory, such as Nix's version (jruby#6608). Instead we detect modules using two other mechanisms that should reliably work on all modularized Java runtimes: * Check for JAVA_HOME/lib/modules, generated by the jlink process. * Check for a JAVA_HOME/release file with a MODULES entry. See https://stackoverflow.com/questions/66601929 for a summary of detection mechanism. Fixes jruby#6608. jruby-launcher will need a matching update.

headius · 2021-03-12T19:19:07Z

I have pushed #6615 to address this after a very helpful thread on Twitter: https://twitter.com/headius/status/1370173969634975746

The jmods directory was a poor way to detect modules because it may be relocated or omitted by a distribution, and doesn't appear in JRE distributions at all (hence they cannot be used to jlink a new runtime, the primary purpose of jmods).

The new logic will use the presence of a lib/modules file, generated by jlink, or the presence of a MODULES entry in the release file shipped with all OpenJDK-derived JDKs and JREs.

@fzakaria Could you give the updated jruby.bash from the PR a test in your environment?

There is a third more complex but fairly robust mechanism we could add: running java -Xinternalversion and parsing the version number... but I would like to avoid having to do that.

Detecting module support based on the existence of the jmods directory was fragile for at least two reasons: * The jmods directory is only present in some builds of the JDK (not JRE) that can be used to jlink new runtime environments. * Some distributions of the JDK may omit or relocate this directory, such as Nix's version (jruby#6608). Instead we detect modules using two other mechanisms that should reliably work on all modularized Java runtimes: * Check for JAVA_HOME/lib/modules, generated by the jlink process. * Check for a JAVA_HOME/release file with a MODULES entry. See https://stackoverflow.com/questions/66601929 for a summary of detection mechanism. Fixes jruby#6608. jruby-launcher will need a matching update.

This mirrors the changes done in jruby/jruby#6615 to fix jruby/jruby#6608.

headius · 2021-03-15T19:32:02Z

This has been fixed on 9.2 and master (9.3) branches as well as in jruby-launcher 1.1.14.

fzakaria mentioned this issue Mar 12, 2021

jruby: Fix JAVA_HOME to support proper jmod support NixOS/nixpkgs#116009

Merged

10 tasks

headius mentioned this issue Mar 12, 2021

Detect modules more robustly #6615

Merged

headius closed this as completed in jruby/jruby-launcher@f995999 Mar 15, 2021

headius added a commit to jruby/jruby-launcher that referenced this issue Mar 15, 2021

Check for modules using /lib/modules and /release

88e3c96

This mirrors the changes done in jruby/jruby#6615 to fix jruby/jruby#6608.

headius added this to the JRuby 9.2.17.0 milestone Mar 15, 2021

headius mentioned this issue Mar 15, 2021

Update Windows jruby.exe for module detection #6617

Closed

Java::JavaLang::NoClassDefFoundError (java/sql/Date) #6608

Java::JavaLang::NoClassDefFoundError (java/sql/Date) #6608

fzakaria commented Mar 11, 2021 •

edited

headius commented Mar 11, 2021

fzakaria commented Mar 11, 2021

headius commented Mar 11, 2021

fzakaria commented Mar 11, 2021

headius commented Mar 11, 2021

headius commented Mar 11, 2021

headius commented Mar 11, 2021 •

edited

fzakaria commented Mar 12, 2021 •

edited

fzakaria commented Mar 12, 2021

fzakaria commented Mar 12, 2021

headius commented Mar 12, 2021

fzakaria commented Mar 12, 2021 •

edited

headius commented Mar 12, 2021

headius commented Mar 15, 2021

Java::JavaLang::NoClassDefFoundError (java/sql/Date) #6608

Java::JavaLang::NoClassDefFoundError (java/sql/Date) #6608

Comments

fzakaria commented Mar 11, 2021 • edited

headius commented Mar 11, 2021

fzakaria commented Mar 11, 2021

headius commented Mar 11, 2021

fzakaria commented Mar 11, 2021

headius commented Mar 11, 2021

headius commented Mar 11, 2021

headius commented Mar 11, 2021 • edited

fzakaria commented Mar 12, 2021 • edited

fzakaria commented Mar 12, 2021

fzakaria commented Mar 12, 2021

headius commented Mar 12, 2021

fzakaria commented Mar 12, 2021 • edited

headius commented Mar 12, 2021

headius commented Mar 15, 2021

fzakaria commented Mar 11, 2021 •

edited

headius commented Mar 11, 2021 •

edited

fzakaria commented Mar 12, 2021 •

edited

fzakaria commented Mar 12, 2021 •

edited