Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Native threads are nil #11450

Closed
amitavmohanty01 opened this issue Dec 17, 2019 · 2 comments
Closed

Native threads are nil #11450

amitavmohanty01 opened this issue Dec 17, 2019 · 2 comments
Assignees
@kares

Comments

@amitavmohanty01
Copy link
Contributor

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For security vulnerabilities please only send reports to security@elastic.co.
See https://www.elastic.co/community/security for more information.

Logstash Plugins are located in a different organization: https://github.com/logstash-plugins. For bugs on specific Logstash plugins, for example, if Redis Output has a defect, please open it in the respective Redis Output repository.

For all general issues, please provide the following details for fast resolution:

  • Version: 7.4.2
  • Operating System: Cent OS 7
  • Steps to Reproduce:
  1. start the service

Issue:

Dec 04 05:17:30 elklogstash04.dvb400.service-now.com systemd[1]: Started logstash.
Dec 04 05:17:41 elklogstash04.dvb400.service-now.com logstash[29134]: Thread.exclusive is deprecated, use Thread::Mutex
Dec 04 05:17:44 elklogstash04.dvb400.service-now.com logstash[29134]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: Exception in thread "Ruby-0-Thread-227: :1" Exception in thread "Ruby-0-Thread-226: :1" org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `id' for nil:NilClass
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at usr.share.logstash.logstash_minus_core.lib.logstash.util.get_thread_id(/usr/share/logstash/logstash-core/lib/logstash/util.rb:42)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at usr.share.logstash.logstash_minus_core.lib.logstash.util.thread_info(/usr/share/logstash/logstash-core/lib/logstash/util.rb:61)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at RUBY.(:1)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.jruby.RubyArray.map(org/jruby/RubyArray.java:2584)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at RUBY.(:1)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.PipelineReporterExt.to_hash(org/logstash/execution/PipelineReporterExt.java:122)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.PipelineReporterExt.snapshot(org/logstash/execution/PipelineReporterExt.java:97)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.ShutdownWatcherExt.pipeline_report_snapshot(org/logstash/execution/ShutdownWatcherExt.java:89)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.ShutdownWatcherExt.start(org/logstash/execution/ShutdownWatcherExt.java:151)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `id' for nil:NilClass
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at usr.share.logstash.logstash_minus_core.lib.logstash.util.get_thread_id(/usr/share/logstash/logstash-core/lib/logstash/util.rb:42)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at usr.share.logstash.logstash_minus_core.lib.logstash.util.thread_info(/usr/share/logstash/logstash-core/lib/logstash/util.rb:61)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at RUBY.(:1)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.jruby.RubyArray.map(org/jruby/RubyArray.java:2584)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at RUBY.(:1)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.PipelineReporterExt.to_hash(org/logstash/execution/PipelineReporterExt.java:122)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.PipelineReporterExt.snapshot(org/logstash/execution/PipelineReporterExt.java:97)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.ShutdownWatcherExt.pipeline_report_snapshot(org/logstash/execution/ShutdownWatcherExt.java:89)
Dec 10 06:43:44 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.ShutdownWatcherExt.start(org/logstash/execution/ShutdownWatcherExt.java:151)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: Exception in thread "Ruby-0-Thread-242: :1" org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `id' for nil:NilClass
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at usr.share.logstash.logstash_minus_core.lib.logstash.util.get_thread_id(/usr/share/logstash/logstash-core/lib/logstash/util.rb:42)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at usr.share.logstash.logstash_minus_core.lib.logstash.util.thread_info(/usr/share/logstash/logstash-core/lib/logstash/util.rb:61)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.plugin_threads_info(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:425)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at org.jruby.RubyArray.map(org/jruby/RubyArray.java:2584)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.plugin_threads_info(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:425)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.PipelineReporterExt.to_hash(org/logstash/execution/PipelineReporterExt.java:122)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.PipelineReporterExt.snapshot(org/logstash/execution/PipelineReporterExt.java:97)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.ShutdownWatcherExt.pipeline_report_snapshot(org/logstash/execution/ShutdownWatcherExt.java:89)
Dec 10 06:45:18 elklogstash04.dvb400.service-now.com logstash[29134]: at org.logstash.execution.ShutdownWatcherExt.start(org/logstash/execution/ShutdownWatcherExt.java:151)

It seems the native threads are nil. I don't know what could cause this. Any suggestions?
@amitavmohanty01 amitavmohanty01 mentioned this issue Dec 17, 2019
Closed
@kares
Copy link
Contributor

kares commented Dec 23, 2019

Thanks for the report, during shutdown threads eventually die (and get garbage collected).
Seems like there's some code running after they're gone assuming to extract and thread id.
The thread unwrapping seems unfortunate since it assumes a WeakReference to be present.

@kares
Copy link
Contributor

kares commented Dec 23, 2019

sorry for the confusion - seems this has been happening on start (before shutdown trigger).
would you by any chance be able to reproduce the issue on latest LS 7.5.1 ?

kares added a commit to kares/logstash that referenced this issue Dec 23, 2019
@kares
Fix: (move and) make get_thread_id "nil safe"
88ff298 
necessary since native (Java) thread is kept as a weak ref
so un-wrapping should deal with a potentially GCd instance

resolves elasticGH-11450
kares added a commit to kares/logstash that referenced this issue Dec 23, 2019
@kares
Fix: (move and) make get_thread_id "nil safe"
307e62d 
necessary since native (Java) thread is kept as a weak ref
so un-wrapping should deal with a potentially GCd instance

resolves elasticGH-11450
@kares kares mentioned this issue Dec 23, 2019
Closed
@kares kares self-assigned this Dec 23, 2019
kares added a commit to kares/logstash that referenced this issue Dec 27, 2019
@kares
Fix: (move and) make get_thread_id "nil safe"
c36981d 
necessary since native (Java) thread is kept as a weak ref
so un-wrapping should deal with a potentially GCd instance

resolves elasticGH-11450
elasticsearch-bot pushed a commit that referenced this issue Dec 30, 2019
@kares
Fix: (move and) make get_thread_id "nil safe"
fd36783 
necessary since native (Java) thread is kept as a weak ref
so un-wrapping should deal with a potentially GCd instance

resolves GH-11450

Fixes #11458
elasticsearch-bot pushed a commit that referenced this issue Dec 30, 2019
@kares
Fix: (move and) make get_thread_id "nil safe"
914577b 
necessary since native (Java) thread is kept as a weak ref
so un-wrapping should deal with a potentially GCd instance

resolves GH-11450

Fixes #11458
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kares kares

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix: (move and) make get_thread_id "nil safe" kares/logstash
2 participants
@kares @amitavmohanty01