Upgrading plugins with manticore throws an error and sometimes corrupts installation #4818
Comments
|
I'm on logstash 2.2.2 not sure if related, I just upgraded to 2.5.3 in order to fix the OOM issues. however, I get the following error when running |
|
@jsvd is it possible to change the issue's title to indicate that logstash is actually broken after the plugin update, so to get more attention? |
|
+1 ran into this issue, it's a heartbreaker |
|
I'm just wondering, is it possible that we now have an issue with the new manticore jumping in?, I'm not sure where I just found they started using jar-dependencies. |
|
been investigating a bit more, updating this for the record.
@dave-kong @githubcdr can you guys provide more details of your infra to help debug and reproduce this? will keep digging. |
jsvd
changed the title
|
@dave-kong updated I'm having some trouble replicating this..can you post your Gemfile.lock of a broken logstash ? |
|
@purbon I basically use the excellent "sebp/elk" docker image that I extend with a default "/opt/logstash/bin/plugin update" and some extra stuff not related to this issue. A docker pull of this image combined with a bash session can reproduce this error here. |
|
@githubcdr the thing is we don't really test, or support docker images. Can you validate your very same error with a real machine (or virtual)? especially the part where logstash is not starting anymore. |
|
I've tested in an ec2 instance using RHEL. I installed logstash-2.2.2 rpm and then, using root, I updated the 2 plugins with the command findings:
output from my experiment: |
|
I'm on ubuntu 14.04 (AWS image). Installed ELK according to this https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04. After I ran into this the first time, I apt-get removed logstash (cleaned entire opt/logstash folder), reinstalled logstash via apt-get. configtest works. Then: |
|
cat /opt/logstash/Gemfile.jruby-1.9.lock GEM PLATFORMS DEPENDENCIES |
|
Quick check agains't two versions of manticore 0.5.5 and 0.5.3. |
|
I am wondering if its not related to the ruby-maven cache, when we install a gem that require a jar it will download the jar into |
|
not familiar with maven cache, but the .m2 folder is empty for my root, or default ubuntu user. if that means anything. |
|
ok I can now replicate this consistently after removing my ~/.m2 directory..digging deeper.. |
|
@dave-kong Can you check in /opt/logstash ? |
|
found another hint with @ph: That failure we thought was benign is actually causing the update to not install the jars BUT since the jar installation is built as post install hook, to the logstash plugin manager the plugin is correctly installed. This means that during the second try everything looks "ok" and the command exits sucessfuly, even though the httpclient jars aren't actually there anymore: Looking at why the first update action failed: This seems related to mkristian/jar-dependencies#39, but now occurring in JRuby 1.7.24. |
|
@ph no .m2 folder in /opt/logstash |
|
the error with non-parsable pom is due to the fact that maven did not install the ruby-DSL for maven. then it fails with the above error and then of course there are no files on .m2 folder will see now if I can reproduce the error locally |
|
all I could find now that "bundler" does not call the post_install_hook after installing the gems, i.e. jar-dependencies never even tries to install the jars. this is when I use the steps to reproduce from the initial report. bundler gives me hard time to include diagnostic output . . still digging |
|
@jsvd the problem is that logstash-2.2.0.tar misses some files from the ruby-maven gem: vendor/bundle/jruby/1.9/gems/ruby-maven-3.3.10/.mvn/extensions.xml adding at vendor/bundle/jruby/1.9/gems/ruby-maven-3.3.10/.mvn/extensions.xml fixes the missing jars for me. the biggest thing was getting some debug output visible. |
|
@jsvd my last sentence was a feature request :) |
|
o.O I have no idea why that file isn't there, we didn't exclude anything in the build process. I'm going to investigate that lead, thanks |
|
I see the issue. during packaging we use jruby-1.7.23 :046 > Dir.glob("**/*").find {|f| f.match(/maven.*extensions.xml/) }
=> nil
jruby-1.7.23 :047 > Dir.glob("**/*", File::FNM_DOTMATCH).find {|f| f.match(/maven.*extensions.xml/) }
=> "vendor/bundle/jruby/1.9/gems/ruby-maven-3.3.10/.mvn/extensions.xml" |
|
so I trick you guys with using this hidden file - sorry, was not intended;)
|
|
confirmed including the file fixes the problem: |
|
Fixed in #4845 |
Steps to reproduce:
Sometimes the installation gets corrupted, see more in #4745
The text was updated successfully, but these errors were encountered: