Update to JRuby `9.4` #14861

andsel · 2023-01-30T11:15:38Z

Release notes

What does this PR do?

Redefine space token in LSCL and grammar treetop from _ which would generated methods in the form def _0 (deprecated since 2.7) to sc.
I18n.t method doesn't accept hash as second argument
~~URI.encode -> URI.encode_www_form_component~~
- URI.encode has been replaced with same functionality with URI::Parser.new.escape
YAML.load needs explicit fallback: false to return false when the yaml string is empty (or contains only comments)
JRuby's JavaClass has been removed, now it can use java.lang.Class directly
explicitly require gem thwait to satisfy require "thwait" (In Gemfile.template and logstash-core/logstash-core.gemspec)
fix not args clone to be def clone(*args)
fix Enumeration.each_slice which from Ruby 3.1 is chainable and doesn't return nil. JRuby fixed in Ruby 3.1 support jruby/jruby#7015
Expanded Down.download arguments map ca16bbe
Avoid to pass nil in the list of couples used in Hash[ <list of couples> ] which from Ruby 3.0 generates an ArgumentError
Removed space not allowed between method name and parentheses initialize ( is forbidden. 29b607d
With Ruby 2.7 the Kernel#open doesn't fallback to URI#open, fixed test code that used that to verify open port. e5b70de
Avoid to drop rdoc/ folder from vendored JRuby else bin/logstash -i irb would crash, commit b71f73e

LoadError: no such file to load -- rdoc/version

Why is it important/What is the impact to the user?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files (and/or docker env variables)
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

check previous update Update jruby version to 9.3.4.0 #14114
Ruby change list in 3.1
check about the problem with lib/jni and glibc, PR to fix in LS is Workaround for #14873 #14890. Verify if port to this or remove the original effort if it's resolved in 9.4.x in the meantime.
verify irb works: bin/logstah -i irb

How to test this PR locally

Related issues

Relates [Meta] Upgrade code and plugins to Jruby 9.4 #14802

Use cases

Screenshots

Logs

andsel · 2023-04-17T08:02:46Z

Jenkins test this please

logstash-core/lib/logstash/api/modules/logging.rb

rubyUtils.gradle

jsvd · 2023-06-15T10:52:52Z

rubyUtils.gradle

-    def gemDir = "${projectDir}/vendor/bundle/jruby/2.6.0".toString()
-    jruby.setLoadPaths(["${projectDir}/vendor/bundle/jruby/2.6.0/gems/bundler-2.4.13/lib".toString(), "${projectDir}/vendor/jruby/lib/ruby/stdlib".toString()])
+    def gemDir = "${projectDir}/vendor/bundle/jruby/3.1.0".toString()
+    jruby.setLoadPaths(["${projectDir}/vendor/bundle/jruby/3.1.0/gems/bundler-2.4.13/lib".toString(), "${projectDir}/vendor/jruby/lib/ruby/stdlib".toString()])


either we apply the same change as in #15066, or need to update the path to be bundler-2.4.14, released 3 days ago https://rubygems.org/gems/bundler/versions/2.4.14

I think should use the Bundler shipped with JRuby and avoid bootstrapping one on our own. So, in this perspective, it should include also the changes proposed with #15066. I can rebase this after the other has been merged into main

jsvd · 2023-06-15T10:53:08Z

spec/unit/plugin_manager/pack_installer/local_spec.rb

@@ -67,8 +67,8 @@
      it "install the gems" do
        expect(::Bundler::LogstashInjector).to receive(:inject!).with(be_kind_of(LogStash::PluginManager::PackInstaller::Pack)).and_return([])

-        expect(::LogStash::PluginManager::GemInstaller).to receive(:install).with(/logstash-input-packtest/, anything)
-        expect(::LogStash::PluginManager::GemInstaller).to receive(:install).with(/logstash-input-packtestdep/, anything)
+        expect(::LogStash::PluginManager::GemInstaller).to receive(:install).with(/logstash-input-packtest-/, anything)


why is this change necessary?

The commit that introduces the changes is 0c92eb3.

Without this change, the test fail:

SPEC_OPTS="-fd -P spec/unit/plugin_manager/pack_installer/local_spec.rb" ./gradlew clean :logstash-core:rubyTests --tests org.logstash.RSpecTests

with

Failures: 1) LogStash::PluginManager::PackInstaller::Local when the local file exist when the pack is valid install the gems Failure/Error: expect(::LogStash::PluginManager::GemInstaller).to receive(:install).with(/logstash-input-packtest/, anything) (LogStash::PluginManager::GemInstaller (class)).install(/logstash-input-packtest/, anything) expected: 1 time with arguments: (/logstash-input-packtest/, anything) received: 2 times with arguments: (/logstash-input-packtest/, anything) # ./spec/unit/plugin_manager/pack_installer/local_spec.rb:70:in `block in <main>' # ./lib/bootstrap/rspec.rb:36:in `<main>' Finished in 0.04501 seconds (files took 0.02848 seconds to load) 5 examples, 1 failure Failed examples: rspec ./spec/unit/plugin_manager/pack_installer/local_spec.rb:67 # LogStash::PluginManager::PackInstaller::Local when the local file exist when the pack is valid install the gems org.logstash.RSpecTests > rspecTests[core tests] FAILED java.lang.AssertionError: RSpec test suite `core tests` saw at least one failure. at org.junit.Assert.fail(Assert.java:88) at org.logstash.RSpecTests.rspecTests(RSpecTests.java:72)

Because the reg exp /logstash-input-packtest/ is matched 2 times:

one with logstash-input-packtest...

another with logstash-input-packtestdep...

I wasn't able to understand why previously it worked and now it receives 2 matches. Also previously it had been to fail.

really strange that it worked before and now doesn't. main and this PR are using the same version of rspec, but bundler and jruby are different.

jsvd · 2023-06-15T14:51:01Z

check about the problem with lib/jni and glibc, PR to fix in LS is #14890. Verify if port to this or remove the original effort if it's resolved in 9.4.x in the meantime.

We can remove the workaround:

[root@a03afd0551ce ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
[root@a03afd0551ce ~]# uname -a
Linux a03afd0551ce 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux[root@a03afd0551ce ~]# ldd jruby-9.4.2.0/lib/jni/x86_64-Linux/libjffi-1.2.so
ldd: warning: you do not have execution permission for `jruby-9.4.2.0/lib/jni/x86_64-Linux/libjffi-1.2.so'
jruby-9.4.2.0/lib/jni/x86_64-Linux/libjffi-1.2.so: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by jruby-9.4.2.0/lib/jni/x86_64-Linux/libjffi-1.2.so)
	libc.so.6 => /lib64/libc.so.6 (0x00007fffff9eb000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fffffddc000)
[root@a03afd0551ce ~]# ldd jruby-9.4.3.0/lib/jni/x86_64-Linux/libjffi-1.2.so
ldd: warning: you do not have execution permission for `jruby-9.4.3.0/lib/jni/x86_64-Linux/libjffi-1.2.so'
	libc.so.6 => /lib64/libc.so.6 (0x00007fffff7ec000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fffffddc000)

jsvd · 2023-06-15T15:08:45Z

Then again, maybe not, still broken on aarch64:

❯ docker run -it oraclelinux:7 bash
[root@5db95de36c6b /]# uname -a
Linux 5db95de36c6b 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
[root@5db95de36c6b /]# curl -s https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.2.21.0/jruby-dist-9.2.21.0-bin.tar.gz | tar -zxf -
[root@5db95de36c6b /]# curl -s https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.3.9.0/jruby-dist-9.3.9.0-bin.tar.gz | tar -zxf -
[root@5db95de36c6b /]# curl -s https://repo1.maven.org/maven2/org/jruby/jruby-dist/9.4.3.0/jruby-dist-9.4.3.0-bin.tar.gz | tar -zxf -
[root@5db95de36c6b /]# ldd jruby-9.2.21.0/lib/jni/aarch64-Linux/libjffi-1.2.so
ldd: warning: you do not have execution permission for `jruby-9.2.21.0/lib/jni/aarch64-Linux/libjffi-1.2.so'
	linux-vdso.so.1 =>  (0x0000ffff89de4000)
	libc.so.6 => /lib64/libc.so.6 (0x0000ffff89c1e000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff89db6000)
[root@5db95de36c6b /]# ldd jruby-9.3.9.0/lib/jni/aarch64-Linux/libjffi-1.2.so
ldd: warning: you do not have execution permission for `jruby-9.3.9.0/lib/jni/aarch64-Linux/libjffi-1.2.so'
	linux-vdso.so.1 =>  (0x0000ffff9bd3f000)
	libc.so.6 => /lib64/libc.so.6 (0x0000ffff9bb79000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff9bd11000)
[root@5db95de36c6b /]# ldd jruby-9.4.3.0/lib/jni/aarch64-Linux/libjffi-1.2.so
ldd: warning: you do not have execution permission for `jruby-9.4.3.0/lib/jni/aarch64-Linux/libjffi-1.2.so'
jruby-9.4.3.0/lib/jni/aarch64-Linux/libjffi-1.2.so: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by jruby-9.4.3.0/lib/jni/aarch64-Linux/libjffi-1.2.so)
	linux-vdso.so.1 =>  (0x0000ffffae948000)
	libc.so.6 => /lib64/libc.so.6 (0x0000ffffae763000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffffae91a000)

Which lines up correctly to the comment here: jruby/jruby#7579 (comment)
Since we support aarch64 in all Operating Systems we support x64 in, we need to keep the patch, but only for aarch64.

andsel · 2023-06-20T18:39:15Z

The 🔴 CI errors, once launched locally with

ci/docker_integration_tests.sh split 0

fails to start Logstash:

      using logstash.yml and separate config file
        behaves like it can send 1000 documents to and index from the dlq

org.logstash.integration.RSpecTests > rspecTests STANDARD_ERROR
    /opt/logstash/qa/integration/framework/fixture.rb:59: warning: Passing safe_level with the 2nd argument of ERB.new is deprecated. Do not use it, and specify other arguments as keyword arguments.
    /opt/logstash/qa/integration/framework/fixture.rb:59: warning: Passing trim_mode with the 3rd argument of ERB.new is deprecated. Use keyword argument like ERB.new(str, trim_mode: ...) instead.

org.logstash.integration.RSpecTests > rspecTests STANDARD_OUT
    Starting Logstash: /opt/logstash/build/logstash-8.9.0-SNAPSHOT/bin/logstash ["-f", "/tmp/studtmp-153c90c79eb67c09d0029b9bfd8475923a7762f1960d4ba85e1932b9c78e", "--path.settings", "/tmp/studtmp-ac45a5f8ad2ee6d79594c605d7ec24c3b025edf830e4ad1d16fdbcfa2d80"] (pwd: /opt/logstash/qa/integration)
    Logstash started with PID 3408, using java: bundled java
    Using bundled JDK: /opt/logstash/build/logstash-8.9.0-SNAPSHOT/jdk
    [FATAL] 2023-06-20 18:25:45.455 [main] Logstash - Logstash was unable to start due to an unexpected Gemfile change.
    If you are a user, this is a bug.
    If you are a logstash developer, please try restarting logstash with the `--enable-local-plugin-development` flag set.

Also launching locally fails with same error, and happens that because the gradle build

./gradlew clean installDefaultGems

Fails with

ERROR: Installation Aborted, message: uninitialized constant Bundler::SolveFailure

that should be fixed by #15066

Jruby: 9.3.40 -> 9.4.0.0 Ruby: 2.6 -> 3.1

With Ruby 2.7 was deprecated the usage of `_1` (`_<n>`) identifier for future reseved usage. With Ruby 3.0 the new synthax for "numbered parameters" was used, avoiding the usage of `_n` as method names. Treetop's generated grammar contained `def _n` methods, mainly implied to the usage of `_` symbol in LSCL grammar. This commit rename the `_` rule in the grammar to `cs` and replaces everywhere it's needed. Explanation at cjheath/treetop#49

…bundled with JRuby

…eds fallback: false

…has to be expaded

…ode. Prefer URI::Parser's escape

…de from Errno::ECONNREFUSED to Errno::EBADF

…rsion.rb file that else make Logstash's irb explode

Co-authored-by: João Duarte <jsvd@users.noreply.github.com>

rubyUtils.gradle

Co-authored-by: João Duarte <jsvd@users.noreply.github.com>

Fixes the warning: warning: nested repeat operator '?' and '*' was replaced with '*' in regular expression /\s?*,\s*/ happening at: logstash-core/lib/logstash/instrument/metric_store.rb

The two warnings are: - lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f - lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int

jsvd

LGTM

…rohibited format (#15500) Updates invocations of i18n.t method which are leftovers and missed in the original Ruby 3.1 update PR #14861 Without this, some error reporting logs are hidden by the mismatch of arguments error in translate the error message.

…rohibited format (#15500) Updates invocations of i18n.t method which are leftovers and missed in the original Ruby 3.1 update PR #14861 Without this, some error reporting logs are hidden by the mismatch of arguments error in translate the error message. (cherry picked from commit 90964fb)

…rohibited format (#15500) (#15501) Updates invocations of i18n.t method which are leftovers and missed in the original Ruby 3.1 update PR #14861 Without this, some error reporting logs are hidden by the mismatch of arguments error in translate the error message. (cherry picked from commit 90964fb) Co-authored-by: Andrea Selva <selva.andre@gmail.com>

andsel self-assigned this Jan 30, 2023

andsel mentioned this pull request Jan 30, 2023

[Meta] Upgrade code and plugins to Jruby 9.4 #14802

Closed

andsel changed the title ~~Fix/updates to jruby 9 4~~ Update to JRuby 9.4 Jan 30, 2023

roaksoax added the status:work-in-progress label Jan 30, 2023

andsel added the JRuby 9.4 label Jan 31, 2023

andsel force-pushed the fix/updates_to_jruby_9_4 branch from 58dfef7 to 42982c1 Compare February 23, 2023 08:28

andsel force-pushed the fix/updates_to_jruby_9_4 branch from b71f73e to 9966e09 Compare March 23, 2023 08:31

andsel force-pushed the fix/updates_to_jruby_9_4 branch from 4ebb9a8 to 85eceb5 Compare April 12, 2023 08:12

andsel force-pushed the fix/updates_to_jruby_9_4 branch from 85eceb5 to cea997f Compare May 31, 2023 13:10

andsel mentioned this pull request Jun 6, 2023

Fix dictionary definition passed into the spec methods logstash-plugins/logstash-filter-cidr#26

Merged

andsel marked this pull request as ready for review June 14, 2023 15:07

jsvd self-requested a review June 14, 2023 15:19

roaksoax added status:needs-review and removed status:work-in-progress labels Jun 14, 2023

jsvd reviewed Jun 15, 2023

View reviewed changes

andsel force-pushed the fix/updates_to_jruby_9_4 branch from 13b04b1 to dcc798b Compare June 15, 2023 12:37

andsel requested a review from jsvd June 15, 2023 13:16

roaksoax linked an issue Jun 22, 2023 that may be closed by this pull request

Switch Logstash to JRuby 9.4 #15113

Closed

andsel added 8 commits June 26, 2023 23:17

Updates all JRuby and ruby versions references

4c618bc

Jruby: 9.3.40 -> 9.4.0.0 Ruby: 2.6 -> 3.1

Removed usage of deprecated API

cd7073c

Fixed the other Treetop grammar

5147b38

Added explicitly dependency on 'thwait' gem (ThreadWait) not anymore …

360d1df

…bundled with JRuby

Fixed YAML.load which, to return false on empty string, explicitly ne…

d182694

…eds fallback: false

URL.encode to URL.encode_www_form_component

fcd2a5e

Fixed I18N translate method params, doesn't accept anymore a map, is …

35b1f9a

…has to be expaded

andsel and others added 10 commits June 26, 2023 23:18

Expanded Down.download arguments map

27604f5

Avoid URI.encode_www_form_component which is not like the old URI.enc…

f40a5f4

…ode. Prefer URI::Parser's escape

Avoid nil in the list of couples used to create an Hash

c48b2a0

Space between method and parentesys is forbidde

a844982

User of URI.open instead of Kernel.open and adapted expected error co…

0db9bfe

…de from Errno::ECONNREFUSED to Errno::EBADF

Avoid to remove the rdoc folder from JRuby std which contains rdoc/ve…

3b9cdaf

…rsion.rb file that else make Logstash's irb explode

Updated JRuby to latest 9.4.2.0

365edcf

Updated JRuby to 9.4.3.0

bafd1e7

Respect the -core JRuby jar introduced with elastic#15050

a914f5b

More straightforward call to clean the nil

19b1da5

Co-authored-by: João Duarte <jsvd@users.noreply.github.com>

andsel force-pushed the fix/updates_to_jruby_9_4 branch from e3906bd to 19b1da5 Compare June 26, 2023 21:18

jsvd reviewed Jun 28, 2023

View reviewed changes

rubyUtils.gradle Outdated Show resolved Hide resolved

andsel and others added 3 commits June 28, 2023 11:07

Update rubyUtils to latest JRuby

068a0ba

Co-authored-by: João Duarte <jsvd@users.noreply.github.com>

Fixes a waring in the usage of regular expression

2f0f088

Fixes the warning: warning: nested repeat operator '?' and '*' was replaced with '*' in regular expression /\s?*,\s*/ happening at: logstash-core/lib/logstash/instrument/metric_store.rb

jsvd approved these changes Jun 28, 2023

View reviewed changes

roaksoax added status:approved and removed status:needs-review labels Jun 28, 2023

andsel merged commit 26d1c7c into elastic:main Jun 28, 2023
3 of 4 checks passed

roaksoax removed the status:approved label Jun 28, 2023

jsvd mentioned this pull request Jul 12, 2023

gem dependencies with non-vendored jars fail to download jar during installDefaultGems #15051

Closed

graybrandonpfg mentioned this pull request Sep 21, 2023

Bugfix: s3snssqs.rb cherweg/logstash-input-s3-sns-sqs#85

Merged

This was referenced Oct 25, 2023

Updates i18n.t method calls to avoid deprecated synthax #15500

Merged

Wrong number of arguments for i18n after upgrade to Version 8.10.3 #15407

Closed

github-actions bot mentioned this pull request Oct 25, 2023

Backport PR #15500 to 8.11: Updates i18n.t method calls to avoid deprecated synthax #15501

Merged

1 task

This was referenced Dec 14, 2023

fail to initialize the plugin with wrong number of arguments (given 1, expected 0) logstash-plugins/logstash-codec-netflow#204

Closed

Fixed ArgumentError unable to initialize the plugin with Logstash 8.10+ logstash-plugins/logstash-codec-netflow#205

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to JRuby `9.4` #14861

Update to JRuby `9.4` #14861

andsel commented Jan 30, 2023 •

edited

andsel commented Apr 17, 2023

jsvd Jun 15, 2023

andsel Jun 15, 2023

jsvd Jun 15, 2023

andsel Jun 15, 2023

andsel Jun 15, 2023

jsvd Jun 15, 2023

jsvd commented Jun 15, 2023

jsvd commented Jun 15, 2023

andsel commented Jun 20, 2023

jsvd left a comment

Update to JRuby 9.4 #14861

Update to JRuby 9.4 #14861

Conversation

andsel commented Jan 30, 2023 • edited

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

andsel commented Apr 17, 2023

jsvd Jun 15, 2023

Choose a reason for hiding this comment

andsel Jun 15, 2023

Choose a reason for hiding this comment

jsvd Jun 15, 2023

Choose a reason for hiding this comment

andsel Jun 15, 2023

Choose a reason for hiding this comment

andsel Jun 15, 2023

Choose a reason for hiding this comment

jsvd Jun 15, 2023

Choose a reason for hiding this comment

jsvd commented Jun 15, 2023

jsvd commented Jun 15, 2023

andsel commented Jun 20, 2023

jsvd left a comment

Choose a reason for hiding this comment

Update to JRuby `9.4` #14861

Update to JRuby `9.4` #14861

andsel commented Jan 30, 2023 •

edited