Logstash 7.17.6 occasionally exits during pipeline start up (throws AliasRegistry related exception)

[armband]

This report describes random startup failures in Logstash 7.17.6 / with preceding AliasRegistry exceptions thrown during pipeline startup. I think this is possibly related to #14002 and, if so, may require a backport to 7.17.x

Logstash information:

Please include the following information:

1. Logstash version:
   • logstash 7.17.6
2. Logstash installation source:
   • Expanded from zip archive,
3. How is Logstash being run:
   • Started as a child process of Linux service (native, non-Docker)

Plugins installed:

Using bundled JDK: /data1/logstash/linux/logstash-7.17.6-1.0.6/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-codec-avro (3.3.1)
logstash-codec-cef (6.2.5)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.1)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.0)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.2.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.1.5)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.11.1)
logstash-filter-fingerprint (3.3.2)
logstash-filter-geoip (7.2.12)
logstash-filter-grok (4.4.2)
logstash-filter-http (1.2.1)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.5.0)
logstash-filter-memcached (1.1.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.6)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.1.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.3.1)
logstash-filter-truncate (1.0.5)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.3)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.1.3)
logstash-input-azure_event_hubs (1.4.4)
logstash-input-beats (6.2.6)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (1.1.12)
logstash-input-elasticsearch (4.12.3)
logstash-input-exec (3.4.0)
logstash-input-file (4.4.3)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.4.5)
logstash-input-http_poller (5.1.0)
logstash-input-imap (3.2.0)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-s3 (3.8.3)
logstash-input-snmp (1.3.1)
logstash-input-snmptrap (3.1.0)
logstash-input-sqs (3.1.3)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.6.0)
logstash-input-tcp (6.2.7)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.1)
logstash-integration-elastic_enterprise_search (2.1.2)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.2.6)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (10.9.0)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq (7.3.0)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-cloudwatch (3.0.10)
logstash-output-csv (3.0.8)
logstash-output-elasticsearch (11.4.1)
logstash-output-email (4.1.1)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.2.5)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-s3 (4.3.7)
logstash-output-sns (4.0.8)
logstash-output-sqs (6.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.0.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.0.6)
logstash-patterns-core (4.3.4)

JVM (e.g. java -version):

openjdk version "11.0.16" 2022-07-19
OpenJDK Runtime Environment Temurin-11.0.16+8 (build 11.0.16+8)
OpenJDK 64-Bit Server VM Temurin-11.0.16+8 (build 11.0.16+8, mixed mode)

OS version

"Ubuntu 18.04.2 LTS (4.15.0-193-generic #204-Ubuntu SMP Fri Aug 26 19:20:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux)"

Description of the problem including expected versus actual behavior:
Pipelines fail to start and Logstash exits:

• Observed occasional, random / unpredictable NoSuchElementException and IllegalStateException exceptions thrown during the startup of the Logstash pipelines. These appear to be related to the AliasRegistry. After the exception is thrown Logstash exits.

Steps to reproduce:
The Logstash configuration includes 29 pipelines, using a mixture of JDBC and Http input plugins. Start Logstash, occasionally startup will fail and Logstash will exit.
(The pipeline conf files can be provided if required)

[2022-10-04T07:44:40,244][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-10-04T07:45:22,286][INFO ][org.reflections.Reflections] Reflections took 1163 ms to scan 1 urls, producing 119 keys and 419 values
[2022-10-04T07:45:22,317][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:group_relation, :exception=>"Java::JavaUtil::NoSuchElementException", :message=>"No line found", :backtrace=>["java.base/java.util.Scanner.nextLine(Scanner.java:1651)", "org.logstash.plugins.AliasRegistry$YamlWithChecksum.load(AliasRegistry.java:69)", "org.logstash.plugins.AliasRegistry$YamlWithChecksum.access$100(AliasRegistry.java:64)", "org.logstash.plugins.AliasRegistry$AliasYamlLoader.loadAliasesDefinitionsFromInputStream(AliasRegistry.java:134)", "org.logstash.plugins.AliasRegistry$AliasYamlLoader.loadAliasesDefinitions(AliasRegistry.java:130)", "org.logstash.plugins.AliasRegistry.<init>(AliasRegistry.java:179)", "org.logstash.plugins.factory.PluginFactoryExt.<init>(PluginFactoryExt.java:89)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:73)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "data1.riverbed.NetIM.$2_dot_7_dot_0_3413.logstash.linux.logstash_minus_7_dot_17_dot_3_minus_1_dot_0_dot_5.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/data1/riverbed/NetIM/2.7.0_3413/logstash/linux/logstash-7.17.3-1.0.5/logstash-core/lib/logstash/java_pipeline.rb:47)", "data1.riverbed.NetIM.$2_dot_7_dot_0_3413.logstash.linux.logstash_minus_7_dot_17_dot_3_minus_1_dot_0_dot_5.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/data1/riverbed/NetIM/2.7.0_3413/logstash/linux/logstash-7.17.3-1.0.5/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "data1.riverbed.NetIM.$2_dot_7_dot_0_3413.logstash.linux.logstash_minus_7_dot_17_dot_3_minus_1_dot_0_dot_5.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/data1/riverbed/NetIM/2.7.0_3413/logstash/linux/logstash-7.17.3-1.0.5/logstash-core/lib/logstash/agent.rb:383)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
[2022-10-04T07:45:58,251][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaUtil::NoSuchElementException` for `PipelineAction::Create<group_relation>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/data1/riverbed/NetIM/2.7.0_3413/logstash/linux/logstash-7.17.3-1.0.5/logstash-core/lib/logstash/agent.rb:396:in `block in converge_state'"]}

[2022-08-22T13:44:14,088][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:subinterface_streaming, :exception=>"Java::JavaLang::IllegalStateException", :message=>"zip file closed", :backtrace=>["java.base/java.util.zip.ZipFile.ensureOpen(ZipFile.java:921)", "java.base/java.util.zip.ZipFile.getEntry(ZipFile.java:348)", "java.base/java.util.zip.ZipFile$1.getEntry(ZipFile.java:1143)", "java.base/java.util.jar.JarFile.getEntry0(JarFile.java:586)", "java.base/java.util.jar.JarFile.getEntry(JarFile.java:516)", "java.base/sun.net.www.protocol.jar.URLJarFile.getEntry(URLJarFile.java:131)", "java.base/sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:137)", "java.base/sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:155)", "java.base/java.net.URL.openStream(URL.java:1165)", "java.base/java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:1739)", "org.logstash.plugins.AliasRegistry$AliasYamlLoader.loadAliasesDefinitions(AliasRegistry.java:124)", "org.logstash.plugins.AliasRegistry.<init>(AliasRegistry.java:179)", "org.logstash.plugins.factory.PluginFactoryExt.<init>(PluginFactoryExt.java:89)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:73)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)",