You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Logstash version (e.g. bin/logstash --version)
logstash 8.5.0
Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
yum install logstash
How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
CLI : nohup /usr/share/logstash/bin/logstash -f /root/logstash/exporter/exporter.conf > f.out 2> f.err < /dev/null &
OS Version:
Linux removed 4.18.0-425.3.1.el8.x86_64 #1 SMP Tue Nov 8 14:08:25 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
Expected
Logstash should flush to disk according to the value of flush_interval
Actual
Logstash never flushes to disk, and eventually runs out of memory
Steps to reproduce:
Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
Logstash Elasticsearch input for any index
Logstash CSV output with any fields
Logstash crash due to java.lang.OutOfMemoryError: Java heap space
Current Result:
Logstash crashes, and all the messages processed are gone.
Expected result:
Messages are processed and flushed to disk periodically
Provide logs (if relevant):
f.out (before 8.5)
Using bundled JDK: /usr/share/logstash/jdk
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-02-03 09:49:40.180 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superser' to 'false' to avoid startup errors in future releases.
[INFO ] 2023-02-03 09:49:40.189 [main] runner - Starting Logstash {"logstash.version"=>"8.4.0", "jruby.version"=>"jruby 9.3.6.0 (2.6.8) 2022-06-27 7a2cbcd376OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}
[INFO ] 2023-02-03 09:49:40.191 [main] runner - JVM bootstrap flags: [-Xms4g, -Xmx4g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableAD=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=fle:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-xports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.ools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMD, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opes=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[WARN ] 2023-02-03 09:49:40.369 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-02-03 09:49:41.000 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2023-02-03 09:49:41.496 [Converge PipelineAction::Create<main>] Reflections - Reflections took 63 ms to scan 1 urls, producing 125 keys and 434 value
[INFO ] 2023-02-03 09:49:41.765 [Converge PipelineAction::Create<main>] javapipeline - Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` seting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[INFO ] 2023-02-03 09:49:41.839 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/root/logstash/exporter/exporter.conf"], :thread=>"#<Thread:0x77495840run>"}
[INFO ] 2023-02-03 09:49:42.192 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.35}
[INFO ] 2023-02-03 09:49:42.688 [[main]-pipeline-manager] elasticsearch - ECS compatibility is enabled but `target` option was not specified. This may cause ields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to void potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[INFO ] 2023-02-03 09:49:42.691 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2023-02-03 09:49:42.717 [[main]|input|elasticsearch|slice_0] elasticsearch - Slice starting {:slice_id=>0, :slices=>4}
[INFO ] 2023-02-03 09:49:42.722 [[main]|input|elasticsearch|slice_2] elasticsearch - Slice starting {:slice_id=>2, :slices=>4}
[INFO ] 2023-02-03 09:49:42.729 [[main]|input|elasticsearch|slice_3] elasticsearch - Slice starting {:slice_id=>3, :slices=>4}
[INFO ] 2023-02-03 09:49:42.730 [[main]|input|elasticsearch|slice_1] elasticsearch - Slice starting {:slice_id=>1, :slices=>4}
[INFO ] 2023-02-03 09:49:42.772 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2023-02-03 09:49:45.489 [[main]>worker0] csv - Opening file {:path=>"/root/logstash/exporter/*removed*.csv"}
f.out (after 8.5)
Using bundled JDK: /usr/share/logstash/jdk
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-02-03 09:52:34.364 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases.
[INFO ] 2023-02-03 09:52:34.373 [main] runner - Starting Logstash {"logstash.version"=>"8.5.0", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}
[INFO ] 2023-02-03 09:52:34.375 [main] runner - JVM bootstrap flags: [-Xms4g, -Xmx4g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[WARN ] 2023-02-03 09:52:34.548 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-02-03 09:52:35.179 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2023-02-03 09:52:35.651 [Converge PipelineAction::Create<main>] Reflections - Reflections took 59 ms to scan 1 urls, producing 125 keys and 438 values
[INFO ] 2023-02-03 09:52:36.200 [Converge PipelineAction::Create<main>] javapipeline - Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[INFO ] 2023-02-03 09:52:36.277 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/root/logstash/exporter/exporter.conf"], :thread=>"#<Thread:0x46ac922e run>"}
[INFO ] 2023-02-03 09:52:36.624 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.35}
[INFO ] 2023-02-03 09:52:37.163 [[main]-pipeline-manager] elasticsearch - ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[INFO ] 2023-02-03 09:52:37.185 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2023-02-03 09:52:37.232 [[main]|input|elasticsearch|slice_1] elasticsearch - Slice starting {:slice_id=>1, :slices=>4}
[INFO ] 2023-02-03 09:52:37.235 [[main]|input|elasticsearch|slice_0] elasticsearch - Slice starting {:slice_id=>0, :slices=>4}
[INFO ] 2023-02-03 09:52:37.261 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2023-02-03 09:52:37.270 [[main]|input|elasticsearch|slice_2] elasticsearch - Slice starting {:slice_id=>2, :slices=>4}
[INFO ] 2023-02-03 09:52:37.279 [[main]|input|elasticsearch|slice_3] elasticsearch - Slice starting {:slice_id=>3, :slices=>4}
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid12882.hprof ...
Heap dump file created [6021510285 bytes in 52.002 secs]
[FATAL] 2023-02-03 09:55:35.506 [Agent thread] Logstash - uncaught error (in thread Agent thread)
java.lang.OutOfMemoryError: Java heap space
Additional info:
This issue happens on 8.6 as well as 8.5. Every other version between 7.17 and onwards till 8.5 works without issues.
You can find more info on exporter.conf and everything else here.
The text was updated successfully, but these errors were encountered:
The issue seems to be in the elasticsearch input plugin.
Logstash 8.4 - default version of logstash-input-elasticsearch plugin is 4.14.0
Upgrading just the input plugin to 4.16.0 breaks it.
Logstash information:
Please include the following information:
Logstash version (e.g. bin/logstash --version)
logstash 8.5.0
Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
yum install logstash
How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
CLI :
nohup /usr/share/logstash/bin/logstash -f /root/logstash/exporter/exporter.conf > f.out 2> f.err < /dev/null &
Plugins installed:
OS Version:
Linux removed 4.18.0-425.3.1.el8.x86_64 #1 SMP Tue Nov 8 14:08:25 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
Logstash should flush to disk according to the value of
flush_interval
Logstash never flushes to disk, and eventually runs out of memory
Steps to reproduce:
Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
Current Result:
Logstash crashes, and all the messages processed are gone.
Expected result:
Messages are processed and flushed to disk periodically
Provide logs (if relevant):
Additional info:
This issue happens on 8.6 as well as 8.5. Every other version between 7.17 and onwards till 8.5 works without issues.
You can find more info on
exporter.conf
and everything else here.The text was updated successfully, but these errors were encountered: