Check third-party libraries against JRuby 10

[headius]

We are about to switch master and jruby-head to JRuby 10, but want to try to make sure that third-party libraries testing against jruby-head do not suddenly fail. To that end, we would like to reproduce the CI test runs from key libraries before making the switch official.

Here is the current list of libraries we intend to verify:

• nokogiri
• sequel
• concurrent-ruby
• json
• asciidoctor
• pdf-reader
• optimist

Of these, sequel and concurrent-ruby are also tested in our CI and green, but I include them here as confirmation.

Several gems in stdlib also run tests in our CI, like json, but we should double-check that their CI processes work correctly with 10.

Users interested in testing their libraries against a JRuby 10 snapshot build can use the following links:

• jruby-dist .tar.gz or .zip (full JRuby install): https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-dist/10.0.0.0-SNAPSHOT/
• jruby-complete (all-in-one jar): https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-complete/10.0.0.0-SNAPSHOT/
• jruby-base (base jar, only includes JRuby): https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-base/10.0.0.0-SNAPSHOT/
• jruby-stdlib (all stdlib in a jar): https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-stdlib/10.0.0.0-SNAPSHOT/
• jruby-jars gem (complete + stdlib jars in a gem): attached to latest build at https://github.com/jruby/jruby/actions/workflows/manual-snapshot-publish-21.yml

There's also some preview Dockerfiles for 10 at https://github.com/jruby/docker-jruby. We have not pushed snapshot images mostly because I don't know where to push them (help wanted!)

We welcome results from other libraries, or requests for libraries you'd like to see tested!

[headius]

Nokogiri's base rake test is green against 10 after bfbc534.

[headius]

Three new failures for asciidoctor comparing 10 with 9.4.11.0:

  1) Failure:
Minitest::Test::TestConverter::TestCustomBackends#test_resolves_templates_from_classloader_when_using_JRuby [test/converter_test.rb:348]:
Expected "<div class=\"ulist\">\n<ul>\n<li>\n<p>foo</p>\n</li>\n<li>\n<p>bar</p>\n</li>\n<li>\n<p>baz</p>\n</li>\n</ul>\n</div>" to include "<span class=\"principal\">foo</span>".

  2) Failure:
Minitest::Test::TestInvoker#test_should_show_timezone_as_UTC_if_system_TZ_is_set_to_UTC [test/invoker_test.rb:847]:
Expected false to be truthy.

  3) Failure:
Minitest::Test::TestInvoker#test_should_show_timezone_as_offset_if_system_TZ_is_not_set_to_UTC [test/invoker_test.rb:855]:
Expected false to be truthy.

Unsure if these are just behavior changes from 3.1 to 3.3 or actual regressions.

[headius]

One failure in json appears to be an API incompatibility or compile issue:

Error: test_raw_strings(JSONAdditionTest): Java::JavaLang::NoSuchMethodError: 'org.jruby.RubyString org.jruby.RubyClass.to_s()'
json.ext.GeneratorMethods$RbString.toJsonRawObject(GeneratorMethods.java:160)
json.ext.GeneratorMethods$RbString.to_json_raw(GeneratorMethods.java:130)

[headius]

The json error appears to be due to a dirty build locally. The released gems and a properly-cleaned head build both pass all tests on 10.

[headius]

My mistake, if I force the tests to run against a released json gem, it does break with that same error.

This is due to the change in 5f0367c removing a number of to_s() signatures from core classes. When compiled directly against those types, Java will use the narrowest signature it can find. If that signature later is removed, it will raise this NoMethodError.

[yob]

It's not a key library, but the test suite for pdf-reader is green on a snapshot of v10: yob/pdf-reader#552

[headius]

@yob I think you are too modest... 77M downloads overall and 500k downloads for a month-old release is impressive!

Thanks so much for the extra work to test this and report back!

[Earlopain]

I've been meaning to do some testing but get an error during bundle install. I can reproduce this by simplying running gem install prism:

$ gem install prism
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
Building native extensions. This could take a while...
ERROR:  Error installing prism:
        ERROR: Failed to build gem native extension.

    current directory: /home/user/.rbenv/versions/jruby-dev/lib/ruby/gems/shared/gems/prism-1.3.0/ext/prism
/home/user/.rbenv/versions/jruby-dev/bin/jruby extconf.rb
extconf failedBad file descriptor - /bin/sh

Gem files will remain installed in /home/user/.rbenv/versions/jruby-dev/lib/ruby/gems/shared/gems/prism-1.3.0 for inspection.
Results logged to /home/user/.rbenv/versions/jruby-dev/lib/ruby/gems/shared/extensions/universal-java-21/3.4.0/prism-1.3.0/gem_make.out

gem_make.out doesn't provide any more useful information. I did a sanity test on 9.4 and it works ok there.

I installed the snapshot via rbenv by modifying jruby-dev and pointing it to the full jruby install. Things like irb seem to work, so I probably did that right? OTOH, I get some suspect warnings on seemingly all commands:

$ ruby -v
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
jruby 10.0.0.0-SNAPSHOT (3.4.0) 2025-03-03 64f2346246 OpenJDK 64-Bit Server VM 21.0.6+7 on 21.0.6+7 +indy +jit [x86_64-linux]

[headius]

@Earlopain Ahh yes, it seems the prism gem does not have a version that installs on JRuby at the moment, but the parser is supported. @enebo do you know status here?

[enebo]

@headius @Earlopain prism gem installs for me on both JRuby 9.4 and JRuby 10 (both linux and macos & with dev build and SNAPSHOT). It definitely works as a gem but there is clearly something wrong happening with your env.

How prism works as a gem on JRuby is that it compiles C source and then uses FFI to call into it. In your case, I am guessing the process execution is not working (likely to comple the source). I have seen your specific error too (but I cannot remember why/where):

extconf failedBad file descriptor - /bin/sh

The other thing I am not seeing is:

WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens

This is from ./bin/.jruby.module_opts and each add-opens is failing because it does not see org.jruby.dist as a Java module. Perhaps @headius can think of a reason for that but something feels amiss there. Actually, I think this module problem is likely the key to this as if you do have proper module access then it may not be allowing process execution to occur.

[enebo]

Another thing I tried was deleting the .jruby.java_opts file and things still work with Java 23.

[enebo]

@Earlopain Is it possible you have jruby-complete.jar in your path somewhere? #7964

[Earlopain]

Is it possible you have jruby-complete.jar in your path somewhere?

No, my env looks clean:

irb(main):006> ENV["PATH"].split(":").map { Dir.glob("#{it}/jruby-complete*") }
=> [[], [], [], [], [], [], [], [], [], [], [], []]

Another thing I tried was deleting the .jruby.java_opts file and things still work with Java 23.

Do you mean .jruby.module_opts? That other file is empty for me. If I remove the module file, the warnings disappear but the error does not.

I will see if I can reproduce this in docker somehow.

[Earlopain]

Here's a fugly dockerfile:

FROM ubuntu:24.04

RUN apt-get update && apt-get install curl git build-essential default-jre -y
RUN git clone https://github.com/rbenv/rbenv.git /root/.rbenv && \
    git clone https://github.com/rbenv/ruby-build.git /root/.rbenv/plugins/ruby-build
RUN ["bash", "-c", "/root/.rbenv/bin/rbenv init"]
RUN sed -i 's=https://github.com/ruby/jruby-dev-builder/releases/latest/download/jruby-head-ubuntu-20.04.tar.gz=https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-dist/10.0.0.0-SNAPSHOT/jruby-dist-10.0.0.0-20250303.220740-1-bin.tar.gz=' /root/.rbenv/plugins/ruby-build/share/ruby-build/jruby-dev
RUN ["bash", "-c", "/root/.rbenv/bin/rbenv install jruby-dev"]
RUN ["bash", "-c", "/root/.rbenv/bin/rbenv global jruby-dev"]

If you then run jruby in there, you get the warnings/errors I wrote about. I suspect my installation method is just wrong. I'm not familiar with JRuby and everything I usually use still points to 9.4 so this was the best I came up with.

Edit: The dockerfile works fine if I point at the 9.4.13.0 snapshot.

[headius]

I am guessing the process execution is not working

Oh yeah that would be my guess too now that I look closer.

@Earlopain You should not be getting the "unknown module" warnings if JRuby was installed properly. It seems like it is not actually running from a full JRuby distribution. Can you show steps you used to set it up? What you described above sounds right.

[headius]

RUN sed -i 's=https://github.com/ruby/jruby-dev-builder/releases/latest/download/jruby-head-ubuntu-20.04.tar.gz=https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-dist/10.0.0.0-SNAPSHOT/jruby-dist-10.0.0.0-20250303.220740-1-bin.tar.gz=' /root/.rbenv/plugins/ruby-build/share/ruby-build/jruby-dev

Something's got to be wrong here, I think... if I'm reading right, you use sed to replace the jruby-dev-builder URL with the snapshot URL, and from there it should be able to unpack the same? Given your warnings it really seems like it's not unpacking right, or the contents of jruby-dev-builder tarballs are laid out differently.

[Earlopain]

if I'm reading right, you use sed to replace the jruby-dev-builder URL with the snapshot URL

Yeah, exactly right. I tried with a different snapshot 9.4.13.0 and that seemed to work fine. It may be a bit too early for me to jump into 10.0.0.0 since I'm not at all familiar (I can't tell if what I was doing is incredibly dumb or not).

[headius]

@Earlopain Ok I think this is something we'll want to look into. If you could, can you just try unpacking the snapshot tarball, putting bin in path, and running gem install prism? If that fails at least we can remove rbenv from the equation.

[Earlopain]

Yes, I think this may be something specific to rbenv. If I do it the manual way like you describe I don't run into any obvious issues like that.

[headius]

@Earlopain Aha, that's unfortunate. I'd really like to know what the problem is, but I assume jruby-dev-builder is doing something special to the builds it makes that is different from what we have in the base tarballs. Perhaps you can raise an issue with rbenv or jruby-dev-builder to get more information?

[Earlopain]

I don't think my use-case of patching the build file would be considered "supported". I'll wait until jruby-dev is switched over to do so.

[headius]

JRuby master is now JRuby 10, so any installers that can do a "head" build should be able to pick it up now.

@Earlopain Perhaps rbenv will now install JRuby 10 for jruby-head?

[Earlopain]

https://github.com/ruby/jruby-dev-builder is still pinned to 9.4. That's were rbenv sources it from, once that is unlocked I can give it a proper try

[headius]

@Earlopain Ok! Main thing preventing me from unfreezing that is getting setup-ruby to set Java 21 so users don't have to. Will try to fix that this week.

[Fryguy]

I'm seeing a StackOverflowError in the optimist gem on jruby-head (jruby 10.0.0.0-SNAPSHOT (3.4.2) 2025-03-14 a4e4d7dc4c OpenJDK 64-Bit Server VM 21.0.6+7-LTS on 21.0.6+7-LTS +indy +jit [x86_64-linux]).

Do you want a separate issues for problems like this? Either way, here's the failing action run: https://github.com/ManageIQ/optimist/actions/runs/13865268977/job/38803994038

[headius]

@Fryguy Hey there! Please open a new bug for this and we will link it here however is appropriate. I don't know of any specific reason why you would have a stack overflow so there's likely something broken.

[enebo]

@Fryguy @headius Another obvious thing to notice when running tests is JRuby 10 is much slower when printing dots. We are churning in something leading up to that stack overflow as well. JRuby 9.4 only takes about 5s to run on my machine.

[Fryguy]

Please open a new bug for this and we will link it here however is appropriate.

Done: #8699

[headius]

The optimist issue from #8699 has been resolved and tests are green! Added to the list.

Also, json is now also green after the fix for #8676.

asciidoctor rake test:all has three new failures on 10, but also triggers a lot of chilled string warnings.

  1) Failure:
Minitest::Test::TestConverter::TestCustomBackends#test_resolves_templates_from_classloader_when_using_JRuby [test/converter_test.rb:348]:
Expected "<div class=\"ulist\">\n<ul>\n<li>\n<p>foo</p>\n</li>\n<li>\n<p>bar</p>\n</li>\n<li>\n<p>baz</p>\n</li>\n</ul>\n</div>" to include "<span class=\"principal\">foo</span>".

  2) Failure:
Minitest::Test::TestInvoker#test_should_show_timezone_as_offset_if_system_TZ_is_not_set_to_UTC [test/invoker_test.rb:855]:
Expected false to be truthy.

  3) Failure:
Minitest::Test::TestInvoker#test_should_show_timezone_as_UTC_if_system_TZ_is_set_to_UTC [test/invoker_test.rb:847]:
Expected false to be truthy.

[headius]

The last two asciidoctor failures above also fail on CRuby 3.4, so those probably aren't valid. The template failure should be investigated, though.