Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to install custom template to aws elasticsearch #101

Closed
jjnajera opened this issue Sep 26, 2018 · 16 comments
Closed

Failing to install custom template to aws elasticsearch #101

jjnajera opened this issue Sep 26, 2018 · 16 comments

Comments

@jjnajera
Copy link

@jjnajera jjnajera commented Sep 26, 2018

The new version of the plugin (6.4.0) is failing to install my custom template to my elasticsearch in AWS. I'm using a docker container with logstash, version 6.3.2), installed. I get the failing error message:

[2018-09-26T01:16:31,260][INFO ][logstash.outputs.elasticsearch] Installing amazon_es template to _template/logstash
[2018-09-26T01:16:31,303][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<URI::InvalidComponentError: bad component(expected absolute path component): _template/logstash>}
[2018-09-26T01:16:31,312][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"bad component(expected absolute path component): _template/logstash", :class=>"URI::InvalidComponentError", :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/generic.rb:769:in `check_path'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/generic.rb:817:in `path='", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/generic.rb:191:in `initialize'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/generic.rb:136:in `build'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/uri/http.rb:63:in `build'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:96:in `perform_request'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:in `perform_request_to_url'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:278:in `block in perform_request'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:373:in `with_connection'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:277:in `perform_request'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:285:in `block in Pool'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:365:in `template_put'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:88:in `template_install'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/template_manager.rb:21:in `install'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/template_manager.rb:9:in `install_template'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:118:in `install_template'", "/logstash-6.4.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:49:in `block in install_template_after_successful_connection'"]}

I have also tried using version 6.4.1 of logstash and still get the same error. I don't know if the path used to install the template should be /_template/#{name} instead of _template/#{name} in the http_client.rb file (line 363).

@xyq164202

This comment has been minimized.

Copy link
Contributor

@xyq164202 xyq164202 commented Sep 26, 2018

According to the error message, it should use absolute path. What's your config file looks like?

@jjnajera

This comment has been minimized.

Copy link
Author

@jjnajera jjnajera commented Sep 27, 2018

My config file looks like this for the output:

amazon_es {
      hosts => ["<removed>.us-east-1.es.amazonaws.com"]
      region => "us-east-1"
      index => "logstash-exchange"
      template => "/logstash-6.3.2/templates/template-es6.json"
}

I use my IAM role to get my AWS id and secret

@xyq164202

This comment has been minimized.

Copy link
Contributor

@xyq164202 xyq164202 commented Sep 28, 2018

Hi jjnajera:
For the template path, can you try the absolute path which form root to your template directory path?

@jjnajera

This comment has been minimized.

Copy link
Author

@jjnajera jjnajera commented Sep 28, 2018

The path to the template in my config file is already the absolute path. The plugin is able to locate my template file, but fails to install it in my elasticsearch instance because I think the URI that is being built is wrong. I notice in the http_client file for the template_exists function that it has the path defined as /_template/#{name} where as the template_put function has the path defined as _template/#{name}.

I changed the path for template_put function to /_template/#{name} and it fixed my error.

@xyq164202

This comment has been minimized.

Copy link
Contributor

@xyq164202 xyq164202 commented Sep 28, 2018

Hi jjnajera:
Thanks for your feedback, will check the http_client function.

@xyq164202 xyq164202 closed this Sep 28, 2018
@jstop

This comment has been minimized.

Copy link

@jstop jstop commented Oct 1, 2018

Not sure this has been fixed. I just installed today and ran into this error. Had to manually update the package with fix from @jjnajera.

@krtxnsln

This comment has been minimized.

Copy link

@krtxnsln krtxnsln commented Oct 2, 2018

thanks @jjnajera that suggestion worked for me as well!

@xyq164202 xyq164202 reopened this Oct 2, 2018
adamgoucher added a commit to adamgoucher/logstash-output-amazon_es that referenced this issue Oct 3, 2018
Fixes awslabs#101
@alexandernajafi

This comment has been minimized.

Copy link

@alexandernajafi alexandernajafi commented Oct 9, 2018

I install the plugin using logstash-plugin install logstash-output-amazon_es. How can I get this fix in? I can't run the plugin currently.

@xyq164202 xyq164202 closed this in 4a00802 Oct 12, 2018
@musabqamri123

This comment has been minimized.

Copy link

@musabqamri123 musabqamri123 commented Oct 29, 2018

The path to the template in my config file is already the absolute path. The plugin is able to locate my template file, but fails to install it in my elasticsearch instance because I think the URI that is being built is wrong. I notice in the http_client file for the template_exists function that it has the path defined as /_template/#{name} where as the template_put function has the path defined as _template/#{name}.

I changed the path for template_put function to /_template/#{name} and it fixed my error.

Did u build the plugin on your own ?

i still see this issue with logstash 6.4.2 & plugin 6.4.0

@aMozejko1

This comment has been minimized.

Copy link

@aMozejko1 aMozejko1 commented Nov 9, 2018

had the same issue but fixed by updating:

/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb

change:

path = "_template/#{name}"

to be:

path = "/_template/#{name}"

in the template_put function.

Alien2150 added a commit to Alien2150/logstash-output-amazon_es that referenced this issue Nov 22, 2018
Bump bugfix for Issues: 
* awslabs#105
* awslabs#101
@mikimichaeli

This comment has been minimized.

Copy link

@mikimichaeli mikimichaeli commented Dec 11, 2018

ANy update on when this is coming out? I am also experiencing this issue with logstash-output-amazon_es version 6.4.0

@rb62680

This comment has been minimized.

Copy link

@rb62680 rb62680 commented Feb 6, 2019

6.4.1 hasn't been pushed yet: https://rubygems.org/gems/logstash-output-amazon_es

Any update?

@pingram3030

This comment has been minimized.

Copy link

@pingram3030 pingram3030 commented Mar 29, 2019

Programmatic hack for future self:

for es_file in $(find /usr/share/logstash/ | grep "amazon_es/http_client.rb"); do
    sed -i -e "s/path \= \"_template/path \= \"\/_template/" ${es_file}
done

puppet code:

  exec { 'fix-es-plugin':
    command => 'bash -c \'for es_file in $(find /usr/share/logstash/ | grep "amazon_es/http_client.rb"); do sed -i -e "s/path \= \"_template/path \= \"\/_template/" ${es_file}; done\'',
    path    => ['/usr/bin', '/usr/sbin']
  }

This is surgical enough to only make the change if the prefixed / is not present. /me dips hat to sed.

@malpani

This comment has been minimized.

Copy link
Contributor

@malpani malpani commented Apr 10, 2019

I released 6.4.1 today which has the patch. Can you please let me know if this is still causing trouble

@PurplePixie-Mpk

This comment has been minimized.

Copy link

@PurplePixie-Mpk PurplePixie-Mpk commented May 24, 2019

@malpani I'm facing the same issue with 6.4.1 and 6.4.2

@suveerrao

This comment has been minimized.

Copy link

@suveerrao suveerrao commented Jun 13, 2019

@jjnajera I tried fix with ES=6.5.0 , LS=6.5.0 and logstash-output-amazon_es = 6.4.0 & 6.4.1 but its the same error. can you please provide the combination that worked for you. I am trying to upgrade our ES=5.5 to 6+ version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.