Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upAllow Embedded ElasticSearch instance to install plugins like cloud-aws, cloud-gce, etc. #1328
Comments
shurane
changed the title from
Allow Embedded ElasticSearch instance to install plugins
to
Allow Embedded ElasticSearch instance to install plugins like cloud-aws, cloud-gce, etc.
May 6, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
colinsurprenant
May 10, 2014
Contributor
thanks for documenting your workaround for this. running embedded elasticseach is typically only meant for local quick & easy prototyping and testing, it wasn't mean for connecting to a cluster or even installing plugins. I suggest you install an external elasticsearch instance and use it to install your plugins and connect to your cluster. Let me know if this makes sense.
|
thanks for documenting your workaround for this. running embedded elasticseach is typically only meant for local quick & easy prototyping and testing, it wasn't mean for connecting to a cluster or even installing plugins. I suggest you install an external elasticsearch instance and use it to install your plugins and connect to your cluster. Let me know if this makes sense. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
Will go ahead and close this. |
colinsurprenant
closed this
May 10, 2014
colinsurprenant
added
the
wontfix
label
May 10, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shurane
May 13, 2014
Contributor
So Logstash uses ES to connect to a cluster anyhow. I'm talking about:
output {
elasticsearch { cluster => "logstash" }
stdout { codec => rubydebug }
}
That relies on the ES that's bundled with Logstash, as far as I can tell.
I'm not sure how else Logstash is supposed to connect to an ES cluster on AWS or GCE.
Although, I know that @imperialwicket from IRC got it up and running another way, though.
|
So That relies on the ES that's bundled with Logstash, as far as I can tell. I'm not sure how else Logstash is supposed to connect to an ES cluster on AWS or GCE. Although, I know that @imperialwicket from IRC got it up and running another way, though. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
colinsurprenant
May 15, 2014
Contributor
Ok, there are 2 things here:
- starting/using an embedded ES server. A local ES server will be started locally and data can be indexed in this local ES.
- logstash using the ES libraries (known as the node or transport protocols) to communicate with a remote ES server.
when using:
output {
elasticsearch {
host => "some_host"
cluster => "logstash"
}
}
logstash will send data to a ES server running on some_host using the node protocol (which is the default, using port 9300). You can also use the http protocol to talk to a remote ES server using the protocol => "http" option.
Does this make sense?
|
Ok, there are 2 things here:
when using: logstash will send data to a ES server running on some_host using the node protocol (which is the default, using port 9300). You can also use the http protocol to talk to a remote ES server using the Does this make sense? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shurane
May 16, 2014
Contributor
Alright, that does make more sense. The embedded ES server uses the same ES libraries that are used for the node or transport protocols when logstash communicates to an ES cluster, correct? Thanks, clears up a bit.
I think what I was trying to say is that with an ES cluster on AWS EC2, logstash cannot discover the cluster using the ES libraries, as by default, the ES cluster on AWS EC2 discovers each other not with multicast/unicast/IP addresses, but using the cloud-aws plugin.
I hope that clears things up a bit.
|
Alright, that does make more sense. The embedded ES server uses the same ES libraries that are used for the node or transport protocols when logstash communicates to an ES cluster, correct? Thanks, clears up a bit. I think what I was trying to say is that with an ES cluster on AWS EC2, logstash cannot discover the cluster using the ES libraries, as by default, the ES cluster on AWS EC2 discovers each other not with multicast/unicast/IP addresses, but using the cloud-aws plugin. I hope that clears things up a bit. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
colinsurprenant
May 17, 2014
Contributor
That's right, logstash includes the ES jars which are used either to spin an embedded ES server or for providing the node/transport protocol.
For speaking to your AWS cluster, you should be able to connect with both node/transport or http, given you expose the correct tcp ports (by default 9200 for http and 9300 for node/transport). Unless you have particular reason for using node/transport, I'd suggest you use http when connecting to a remote cluster.
Let me know if that help.
|
That's right, logstash includes the ES jars which are used either to spin an embedded ES server or for providing the node/transport protocol. For speaking to your AWS cluster, you should be able to connect with both node/transport or http, given you expose the correct tcp ports (by default 9200 for http and 9300 for node/transport). Unless you have particular reason for using node/transport, I'd suggest you use http when connecting to a remote cluster. Let me know if that help. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
joekiller
Jul 11, 2014
Contributor
For anyone trying to run the cloud-aws plugin I solved it via: elastic/elasticsearch-cloud-aws#71 (comment)
|
For anyone trying to run the cloud-aws plugin I solved it via: elastic/elasticsearch-cloud-aws#71 (comment) |
colinsurprenant
referenced this issue
Jul 11, 2014
Closed
Logstash 1.4 + cloud-aws 2.1.0 + elasticsearch 1.1 #71
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jlintz
Aug 1, 2014
Just ran into this issue as well. Would really love the ability to easily install the cloud-aws plugin to have it connect to my existing ES cluster. Was unable to find any documentation on how to go about doing so, and this github issue was the only reference I've found
jlintz
commented
Aug 1, 2014
|
Just ran into this issue as well. Would really love the ability to easily install the cloud-aws plugin to have it connect to my existing ES cluster. Was unable to find any documentation on how to go about doing so, and this github issue was the only reference I've found |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
vdbrooks
Aug 1, 2014
+1 for @jlintz comment. I am implementing this today on AWS and also need the ability to install the cloud-aws plugin via Logstash. Have some hope that @joekiller solution will be the answer.
vdbrooks
commented
Aug 1, 2014
|
+1 for @jlintz comment. I am implementing this today on AWS and also need the ability to install the cloud-aws plugin via Logstash. Have some hope that @joekiller solution will be the answer. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jlintz
Aug 1, 2014
@vdbrooks I ended up just setting up a non-data ES instance with the cloud-aws plugin configured and then pointed logstash at localhost to workaround this. It was a bit more straight forward to handle with the puppet modules
jlintz
commented
Aug 1, 2014
|
@vdbrooks I ended up just setting up a non-data ES instance with the cloud-aws plugin configured and then pointed logstash at localhost to workaround this. It was a bit more straight forward to handle with the puppet modules |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
joekiller
Aug 1, 2014
Contributor
The easiest way for me to make this work is to install ElasticSearch 1.1.1 from the binaries, install the cloud-aws plugin via their bin/plugin script and then add to my /etc/sysconfig/logstash entry the following:
LS_JAVA_OPTS="$LS_JAVA_OPTS -Des.path.home=/usr/local/elasticsearch -Des.config=/opt/logstash/elasticsearch.yml"
The LS_JAVA_OPTS is picked up by bin/logstash and appended to the startup. You can modify anything that ElasticSearch is looking for by checking out the following reference: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-dir-layout.html
I just set my es.path.home to the root of the elasticsearch install because that allows it to resolve the {es.home}/plugins path as well which is easier than copying stuff all over the place.
YOU MUST MATCH ES BINARY VERSIONS FOR THIS TO WORK. IE ES 1.1.1 is what Logstash 1.4.2 uses. This all works because logstash's internal ElasticSearch jar file is referenced first so the only jars that are loaded afterwards are those which are not present at startup.
Side note you can see from elasticsearch's bin/plugin file that it is really just a java call that they are executing (https://github.com/elasticsearch/elasticsearch/blob/master/bin/plugin). Someone could feasibly use the jar files in logstash's vendor/jar/elasticsearch/lib directory to make the proper org.elasticsearch.plugins.PluginManager call but I'm not bothering because I'm installing a mirror ES 1.1.1 binary on the machine along with logstash to get the plugin. Different strokes people but in the end, use the java opts to make logstash do your dance.
|
The easiest way for me to make this work is to install ElasticSearch 1.1.1 from the binaries, install the cloud-aws plugin via their bin/plugin script and then add to my /etc/sysconfig/logstash entry the following: The LS_JAVA_OPTS is picked up by I just set my es.path.home to the root of the elasticsearch install because that allows it to resolve the {es.home}/plugins path as well which is easier than copying stuff all over the place. YOU MUST MATCH ES BINARY VERSIONS FOR THIS TO WORK. IE ES 1.1.1 is what Logstash 1.4.2 uses. This all works because logstash's internal ElasticSearch jar file is referenced first so the only jars that are loaded afterwards are those which are not present at startup. Side note you can see from elasticsearch's bin/plugin file that it is really just a java call that they are executing (https://github.com/elasticsearch/elasticsearch/blob/master/bin/plugin). Someone could feasibly use the jar files in logstash's vendor/jar/elasticsearch/lib directory to make the proper org.elasticsearch.plugins.PluginManager call but I'm not bothering because I'm installing a mirror ES 1.1.1 binary on the machine along with logstash to get the plugin. Different strokes people but in the end, use the java opts to make logstash do your dance. |
shurane commentedMay 5, 2014
Request
Expose and document embedded ElasticSearch instance to install plugins (
cloud-aws,cloud-gce, etc) inside Logstash.Motivation
I tried connecting my Logstash node to an ElasticSearch cluster over AWS. I connect the ES nodes in the ES cluster using the cloud-aws plugin. Connecting my LS node took a bit more time, mainly because I couldn't find a way to install the
cloud-awsplugin from Logstash. While the ES nodes connected to each other happily, my Logstash node just refused to connect. Even running a separate local instance of ES connected to the cluster fine, but LS did not communicate to the local instance of ES (usingoutput { elasticsearch { host => localhost cluster => "logstash" } }). I did eventually resolve this issue by rsyncing over the cloud-aws/ plugin intologstash-1.4.0/vendor/jar/elasticsearch-1.0.1/plugins/.Alternatively, I could have just rsynced the
elasticsearch-1.0.1/bin/directory over tologstash-1.4.0/vendor/jar/elasticsearch-1.0.1/bin/, and then install plugins and treat it as if it was a regular version of elasticsearch.My Approach
Some terminal output