It is possible for plugin alias support to emit NPE #13666
Assignees
Labels
Comments
Merged
2 tasks
2 tasks
|
Looking at the stack trace the alias loader encounter the problem when it tries to load the logstash/logstash-core/src/main/java/org/logstash/plugins/AliasRegistry.java Lines 123 to 130 in c64ef88 So it meas that the |
robbavey
added a commit
to robbavey/logstash
that referenced
this issue
Apr 19, 2022
The current implementation of AliasRegistry will create an instance of the Alias Registry for each pipeline, which appears to potentially result in situations, such as in elastic#13996, where multiple pipelines are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when reading the yaml file as the JarFile has been closed after the initial thread completed accessing. This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry. Relates: elastic#13996, elastic#13666
robbavey
added a commit
to robbavey/logstash
that referenced
this issue
Apr 19, 2022
The current implementation of AliasRegistry will create an instance of the Alias Registry for each pipeline, which appears to potentially result in situations, such as in elastic#13996, where multiple pipelines are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when reading the yaml file as the JarFile has been closed after the initial thread completed accessing. This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry. Relates: elastic#13996, elastic#13666
robbavey
added a commit
to robbavey/logstash
that referenced
this issue
Apr 19, 2022
The current implementation of AliasRegistry will create an instance of the Alias Registry for each pipeline, which appears to potentially result in situations, such as in elastic#13996, where multiple pipelines are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when reading the yaml file as the JarFile has been closed after the initial thread completed accessing. This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry. Relates: elastic#13996, elastic#13666
robbavey
added a commit
to robbavey/logstash
that referenced
this issue
Apr 19, 2022
The current implementation of AliasRegistry will create an instance of the Alias Registry for each pipeline, which appears to potentially result in situations, such as in elastic#13996, where multiple pipelines are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when reading the yaml file as the JarFile has been closed after the initial thread completed accessing. This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry. Relates: elastic#13996, elastic#13666
|
Looking around for this I discovered:
The suspect is that loading the aliases definitions from multiple threads could be the problem of concurrent closes of the zipped input stream. Possible solution are:
I think the best is the second one, but before a multithreaded unit test has to be create to proof it's a concurrency problem related to the close of the resource. |
robbavey
added a commit
that referenced
this issue
Apr 26, 2022
* Make AliasRegistry a singleton The current implementation of AliasRegistry will create an instance of the Alias Registry for each pipeline, which appears to potentially result in situations, such as in #13996, where multiple pipelines are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when reading the yaml file as the JarFile has been closed after the initial thread completed accessing. This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry. Relates: #13996, #13666 * Update reflections library to 0.9.12 to avoid multi threading bug Earlier versions of the reflections library used in the plugin registry would use caches on JarUrlConnection, which when closed would also close the jar file for other resources using it, such as the AliasRegistry. This, combined with the fact that the AliasRegistry could be created simultaneously by many threads/pipelines could cause issues during AliasRegistry creation leading to failure to create a pipeline. * Avoid use of URLConnection caching when getting alias yaml resource * Use idiomatic ruby when accessing Java getInstance method Co-authored-by: Andrea Selva <selva.andre@gmail.com>
github-actions bot
pushed a commit
that referenced
this issue
Apr 26, 2022
* Make AliasRegistry a singleton The current implementation of AliasRegistry will create an instance of the Alias Registry for each pipeline, which appears to potentially result in situations, such as in #13996, where multiple pipelines are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when reading the yaml file as the JarFile has been closed after the initial thread completed accessing. This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry. Relates: #13996, #13666 * Update reflections library to 0.9.12 to avoid multi threading bug Earlier versions of the reflections library used in the plugin registry would use caches on JarUrlConnection, which when closed would also close the jar file for other resources using it, such as the AliasRegistry. This, combined with the fact that the AliasRegistry could be created simultaneously by many threads/pipelines could cause issues during AliasRegistry creation leading to failure to create a pipeline. * Avoid use of URLConnection caching when getting alias yaml resource * Use idiomatic ruby when accessing Java getInstance method Co-authored-by: Andrea Selva <selva.andre@gmail.com> (cherry picked from commit 4e77f1b)
|
Yes, I agree. Closing |
robbavey
added a commit
that referenced
this issue
May 6, 2022
* Make AliasRegistry a singleton The current implementation of AliasRegistry will create an instance of the Alias Registry for each pipeline, which appears to potentially result in situations, such as in #13996, where multiple pipelines are simultaneously loading an alias registry from a yaml file in the jar file using getResourceAsStream, with the potential of the first thread closing the jar file underneath subsequent threads, leading to errors when reading the yaml file as the JarFile has been closed after the initial thread completed accessing. This commit changes the AliasRegistry to be a singleton, as is the PluginRegistry. Relates: #13996, #13666 * Update reflections library to 0.9.12 to avoid multi threading bug Earlier versions of the reflections library used in the plugin registry would use caches on JarUrlConnection, which when closed would also close the jar file for other resources using it, such as the AliasRegistry. This, combined with the fact that the AliasRegistry could be created simultaneously by many threads/pipelines could cause issues during AliasRegistry creation leading to failure to create a pipeline. * Avoid use of URLConnection caching when getting alias yaml resource * Use idiomatic ruby when accessing Java getInstance method Co-authored-by: Andrea Selva <selva.andre@gmail.com> (cherry picked from commit 4e77f1b) Co-authored-by: Rob Bavey <rob.bavey@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Encountered this in a CI run of integration tests:
The text was updated successfully, but these errors were encountered: