Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Work scheduler broken in 8.10 BC using JRuby 9.4.3.0 #15282

Closed
yaauie opened this issue Aug 29, 2023 · 1 comment · Fixed by #15369
Closed

Work scheduler broken in 8.10 BC using JRuby 9.4.3.0 #15282

yaauie opened this issue Aug 29, 2023 · 1 comment · Fixed by #15369
Assignees
@yaauie
Labels
bug status:needs-triage

Comments

@yaauie
Copy link
Member

yaauie commented Aug 29, 2023

Logstash information:

Please include the following information:

  1. Logstash version: 8.10 BC2
  2. Logstash installation source: ANY
  3. How is Logstash being run: ANY

Plugins installed: (bin/logstash-plugin list --verbose)

JVM (e.g. java -version):

  1. JVM version (java -version) 11, 17, 20
  2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
  3. Value of the LS_JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

The Concurrent::TimerTask's scheduling thread silently dies when JRuby 9.4.3.0's invokedynamic is enabled after several minutes of running normally. This has been reported upstream in jruby/jruby#7904

This single-threaded executor is responsible for scheduling future invocations of all TimerTask, which is what powers a lot:

  • x-pack monitoring collection
  • system and jvm metrics captures for API consumption
  • flow metric snapshots
  • x-pack geoip database manager
  • s3 output plugin basic functionality
  • http output plugin basic functionality

Steps to reproduce:

  1. Run Logstash with debug-level logging enabled for several minutes 
    ${LOGSTASH_HOME}/bin/logstash --log.level debug -e 'filter {}'
  2. observe that the debug logging of things like flow metrics collection ceases wile Logstash itself continues to run fairly-normally
yaauie added a commit to yaauie/logstash that referenced this issue Aug 29, 2023
@yaauie
deps: downgrade java, keep updated gem dependencies
ae05467 
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.

Upstream bug: jruby/jruby#7904

Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.

We also remove a pinned-dependency on `racc` that is no longer relevant.

Resolves: elastic#15282
yaauie added a commit to yaauie/logstash that referenced this issue Aug 29, 2023
@yaauie
deps: downgrade java, keep updated gem dependencies
0058136 
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.

Upstream bug: jruby/jruby#7904

Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.

We also remove a pinned-dependency on `racc` that is no longer relevant.

Resolves: elastic#15282
@yaauie yaauie mentioned this issue Aug 29, 2023
Merged
2 tasks
yaauie added a commit to yaauie/logstash that referenced this issue Aug 29, 2023
@yaauie
deps: downgrade jruby, keep updated default-gem dependencies
794fd9a 
By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.

Upstream bug: jruby/jruby#7904

Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.

We also remove a pinned-dependency on `racc` that is no longer relevant.

Resolves: elastic#15282
yaauie added a commit that referenced this issue Aug 29, 2023
@yaauie @andsel
deps: downgrade jruby, keep updated default-gem dependencies (#15283)
df250c4 
* deps: downgrade jruby, keep updated default-gem dependencies

By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.

Upstream bug: jruby/jruby#7904

Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.

We also remove a pinned-dependency on `racc` that is no longer relevant.

Resolves: #15282

* Imported the licenses for some gems

- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri

* specs: avoid mocking global ::Gem::Dependency::new

* build: remove redundanct dependsOn declaration

* deps: notice use of ffi-binary-libfixposix via Ruby license

this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available

---------

Co-authored-by: andsel <selva.andre@gmail.com>
@yaauie
Copy link
Member Author

yaauie commented Aug 29, 2023

Work on the 8.10 branch has been merged, I am leaving this issue open until:

  • DRA includes fix
  • relevant work is forward-ported to main

yaauie added a commit to yaauie/logstash that referenced this issue Oct 3, 2023
@yaauie @andsel
deps: downgrade jruby, keep updated default-gem dependencies (elastic…
35d59e1 
…#15283)

forward-ports non-release-branch components of elastic#15283 to `main`

* deps: downgrade jruby, keep updated default-gem dependencies

By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.

Upstream bug: jruby/jruby#7904

Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.

We also remove a pinned-dependency on `racc` that is no longer relevant.

Resolves: elastic#15282

* Imported the licenses for some gems

- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri

* specs: avoid mocking global ::Gem::Dependency::new

* build: remove redundanct dependsOn declaration

* deps: notice use of ffi-binary-libfixposix via Ruby license

this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available

---------

Co-authored-by: andsel <selva.andre@gmail.com>
@yaauie yaauie mentioned this issue Oct 3, 2023
Merged
2 tasks
mashhurs pushed a commit that referenced this issue Oct 3, 2023
@yaauie @andsel
deps: downgrade jruby, keep updated default-gem dependencies (forward…
70081bb 
…-port #15283) (#15369)

* deps: downgrade jruby, keep updated default-gem dependencies (#15283)

forward-ports non-release-branch components of #15283 to `main`

* deps: downgrade jruby, keep updated default-gem dependencies

By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.

Upstream bug: jruby/jruby#7904

Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.

We also remove a pinned-dependency on `racc` that is no longer relevant.

Resolves: #15282

* Imported the licenses for some gems

- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri

* specs: avoid mocking global ::Gem::Dependency::new

* build: remove redundanct dependsOn declaration

* deps: notice use of ffi-binary-libfixposix via Ruby license

this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available

---------

Co-authored-by: andsel <selva.andre@gmail.com>

* deps: add license notices for gems moved from default to bundled

---------

Co-authored-by: andsel <selva.andre@gmail.com>
github-actions bot pushed a commit that referenced this issue Oct 3, 2023
@yaauie @logstashmachine
deps: downgrade jruby, keep updated default-gem dependencies (forward…
bb85147 
…-port #15283) (#15369)

* deps: downgrade jruby, keep updated default-gem dependencies (#15283)

forward-ports non-release-branch components of #15283 to `main`

* deps: downgrade jruby, keep updated default-gem dependencies

By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.

Upstream bug: jruby/jruby#7904

Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.

We also remove a pinned-dependency on `racc` that is no longer relevant.

Resolves: #15282

* Imported the licenses for some gems

- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri

* specs: avoid mocking global ::Gem::Dependency::new

* build: remove redundanct dependsOn declaration

* deps: notice use of ffi-binary-libfixposix via Ruby license

this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available

---------

Co-authored-by: andsel <selva.andre@gmail.com>

* deps: add license notices for gems moved from default to bundled

---------

Co-authored-by: andsel <selva.andre@gmail.com>
(cherry picked from commit 70081bb)
@github-actions github-actions bot mentioned this issue Oct 3, 2023
Merged
2 tasks
mashhurs pushed a commit that referenced this issue Oct 3, 2023
@github-actions @yaauie
deps: downgrade jruby, keep updated default-gem dependencies (forward…
13309ad 
…-port #15283) (#15369) (#15370)

* deps: downgrade jruby, keep updated default-gem dependencies (#15283)

forward-ports non-release-branch components of #15283 to `main`

* deps: downgrade jruby, keep updated default-gem dependencies

By downgrading JRuby to 9.4.2.0 we avoid the silent global crash of the
scheduler backing `Concurrent::TimerTask` that occurs when Jruby 9.4.3.0's
invokedynamic promotes a method to run natively, incorrectly.

Upstream bug: jruby/jruby#7904

Along with the downgrade of JRuby itself to 9.4.2.0, we cherry-pick the
updates to gems that were included in the latest JRuby 9.4.3.0 to ensure
we don't back out relevant fixes to stdlib.

We also remove a pinned-dependency on `racc` that is no longer relevant.

Resolves: #15282

* Imported the licenses for some gems

- cgi
- date
- ffi-binary-libfixposix
- io-console
- net-http
- net-protocol
- reline
- time
- timeout
- uri

* specs: avoid mocking global ::Gem::Dependency::new

* build: remove redundanct dependsOn declaration

* deps: notice use of ffi-binary-libfixposix via Ruby license

this gem is tri-licensed `Ruby` / `EPL-2.0` / `LGPL-2.1-or-later` and
the Ruby license is preferred to EPL when available

---------

Co-authored-by: andsel <selva.andre@gmail.com>

* deps: add license notices for gems moved from default to bundled

---------

Co-authored-by: andsel <selva.andre@gmail.com>
(cherry picked from commit 70081bb)

Co-authored-by: Ry Biesemeyer <yaauie@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yaauie yaauie

Labels
bug status:needs-triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

deps: downgrade jruby, keep updated default-gem dependencies (forward-port #15283) yaauie/logstash
1 participant
@yaauie