From e3443dc1bf951dff13bb6d3b0ab6cfb9e334a35a Mon Sep 17 00:00:00 2001 From: Nick Allen Date: Tue, 3 Jul 2018 08:43:17 -0400 Subject: [PATCH 1/3] Numbered the X-pack instructions to make them easier to follow --- metron-deployment/Kerberos-manual-setup.md | 392 ++++++++++----------- 1 file changed, 196 insertions(+), 196 deletions(-) diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md index ecda858329..11276dcdc1 100644 --- a/metron-deployment/Kerberos-manual-setup.md +++ b/metron-deployment/Kerberos-manual-setup.md @@ -538,203 +538,203 @@ In order to correct this, you should: X-Pack ------ -First, stop the random_access_indexing topology through the Storm UI or from the CLI, e.g. - -``` -storm kill random_access_indexing -``` - Here are instructions for enabling X-Pack with Elasticsearch and Kibana: https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html -You need to be sure to add the appropriate username and password for Elasticsearch and Kibana to enable external connections from Metron components. e.g. the following will create a user "transport_client_user" with password "changeme" and "superuser" credentials. - -``` -sudo /usr/share/elasticsearch/bin/x-pack/users useradd transport_client_user -p changeme -r superuser -``` - -Once you've picked a password to connect to ES, you need to upload a 1-line file to HDFS with that password in it. Metron will use this file to securely read the password in order to connect to ES securely. - -Here is an example using "changeme" as the password - -``` -echo changeme > /tmp/xpack-password -sudo -u hdfs hdfs dfs -mkdir /apps/metron/elasticsearch/ -sudo -u hdfs hdfs dfs -put /tmp/xpack-password /apps/metron/elasticsearch/ -sudo -u hdfs hdfs dfs -chown metron:metron /apps/metron/elasticsearch/xpack-password -``` - -New settings have been added to configure the Elasticsearch client. By default the client will run as the normal ES prebuilt transport client. If you enable X-Pack you should set the es.client.class as shown below. - -Add the es settings to global.json - -``` -/usr/metron/0.5.1/config/zookeeper/global.json -> - - "es.client.settings" : { - "es.client.class" : "org.elasticsearch.xpack.client.PreBuiltXPackTransportClient", - "es.xpack.username" : "transport_client_user", - "es.xpack.password.file" : "/apps/metron/elasticsearch/xpack-password" - } -``` - -Submit the update to Zookeeper - -``` -$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i METRON_HOME/config/zookeeper/ -z $ZOOKEEPER -``` - -The last step before restarting the topology is to create a custom X-Pack shaded and relocated jar. This is up to you because of licensing restrictions, but here is a sample Maven pom file that should help. - -``` - - - - 4.0.0 - org.elasticsearch - elasticsearch-xpack-shaded - elasticsearch-xpack-shaded - jar - 5.6.2 - - - elasticsearch-releases - https://artifacts.elastic.co/maven - - true - - - false - - - - - - org.elasticsearch.client - x-pack-transport - 5.6.2 - - - com.fasterxml.jackson.dataformat - jackson-dataformat-yaml - - - com.fasterxml.jackson.dataformat - jackson-dataformat-cbor - - - com.fasterxml.jackson.core - jackson-core - - - org.slf4j - slf4j-api - - - org.slf4j - slf4j-log4j12 - - - log4j - log4j - - - org.apache.logging.log4j - log4j-api - - - - - - - - org.apache.maven.plugins - maven-shade-plugin - 2.4.3 - - true - - - - package - - shade - - - - - *:* - - META-INF/*.SF - META-INF/*.DSA - META-INF/*.RSA - - - - - - io.netty - org.apache.metron.io.netty - - - org.apache.logging.log4j - org.apache.metron.logging.log4j - - - - - org.slf4j.impl* - org.slf4j:slf4j-log4j* - - - - - - .yaml - LICENSE.txt - ASL2.0 - NOTICE.txt - - - - - - - - - - - - - - -``` - -Once you've built the elasticsearch-xpack-shaded-5.6.2.jar, it needs to be made available to Storm when you submit the topology. Create a contrib directory for indexing and put the jar file in this directory. - -``` -/usr/metron/0.5.1/indexing_contrib/elasticsearch-xpack-shaded-5.6.2.jar -``` - -Now you can restart the Elasticsearch topology. Note, you should perform this step manually, as follows. +1. First, stop the random_access_indexing topology through the Storm UI or from the CLI, e.g. -``` -$METRON_HOME/bin/start_elasticsearch_topology.sh -``` + ``` + storm kill random_access_indexing + ``` -Once you've performed these steps, you shoud be able to start seeing data in your ES indexes. +1. You need to be sure to add the appropriate username and password for Elasticsearch and Kibana to enable external connections from Metron components. e.g. the following will create a user "transport_client_user" with password "changeme" and "superuser" credentials. + + ``` + sudo /usr/share/elasticsearch/bin/x-pack/users useradd transport_client_user -p changeme -r superuser + ``` + +1. Once you've picked a password to connect to ES, you need to upload a 1-line file to HDFS with that password in it. Metron will use this file to securely read the password in order to connect to ES securely. + + Here is an example using "changeme" as the password + + ``` + echo changeme > /tmp/xpack-password + sudo -u hdfs hdfs dfs -mkdir /apps/metron/elasticsearch/ + sudo -u hdfs hdfs dfs -put /tmp/xpack-password /apps/metron/elasticsearch/ + sudo -u hdfs hdfs dfs -chown metron:metron /apps/metron/elasticsearch/xpack-password + ``` + +1. New settings have been added to configure the Elasticsearch client. By default the client will run as the normal ES prebuilt transport client. If you enable X-Pack you should set the es.client.class as shown below. + + Add the `es.client.settings` to global.json + + ``` + /usr/metron/0.5.1/config/zookeeper/global.json -> + + "es.client.settings" : { + "es.client.class" : "org.elasticsearch.xpack.client.PreBuiltXPackTransportClient", + "es.xpack.username" : "transport_client_user", + "es.xpack.password.file" : "/apps/metron/elasticsearch/xpack-password" + } + ``` + + Submit the update to Zookeeper + + ``` + $METRON_HOME/bin/zk_load_configs.sh -m PUSH -i METRON_HOME/config/zookeeper/ -z $ZOOKEEPER + ``` + +1. The last step before restarting the topology is to create a custom X-Pack shaded and relocated jar. This is up to you because of licensing restrictions, but here is a sample Maven pom file that should help. + + ``` + + + + 4.0.0 + org.elasticsearch + elasticsearch-xpack-shaded + elasticsearch-xpack-shaded + jar + 5.6.2 + + + elasticsearch-releases + https://artifacts.elastic.co/maven + + true + + + false + + + + + + org.elasticsearch.client + x-pack-transport + 5.6.2 + + + com.fasterxml.jackson.dataformat + jackson-dataformat-yaml + + + com.fasterxml.jackson.dataformat + jackson-dataformat-cbor + + + com.fasterxml.jackson.core + jackson-core + + + org.slf4j + slf4j-api + + + org.slf4j + slf4j-log4j12 + + + log4j + log4j + + + org.apache.logging.log4j + log4j-api + + + + + + + + org.apache.maven.plugins + maven-shade-plugin + 2.4.3 + + true + + + + package + + shade + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + io.netty + org.apache.metron.io.netty + + + org.apache.logging.log4j + org.apache.metron.logging.log4j + + + + + org.slf4j.impl* + org.slf4j:slf4j-log4j* + + + + + + .yaml + LICENSE.txt + ASL2.0 + NOTICE.txt + + + + + + + + + + + + + + + ``` + +1. Once you've built the `elasticsearch-xpack-shaded-5.6.2.jar`, it needs to be made available to Storm when you submit the topology. Create a contrib directory for indexing and put the jar file in this directory. + + ``` + /usr/metron/0.5.1/indexing_contrib/elasticsearch-xpack-shaded-5.6.2.jar + ``` + +1. Now you can restart the Elasticsearch topology. Note, you should perform this step manually, as follows. + + ``` + $METRON_HOME/bin/start_elasticsearch_topology.sh + ``` + +Once you've performed these steps, you should be able to start seeing data in your ES indexes. From 1523765ae79f8aefac704fb643a8812b36c4ea73 Mon Sep 17 00:00:00 2001 From: Nick Allen Date: Tue, 3 Jul 2018 08:50:49 -0400 Subject: [PATCH 2/3] Added description of common problem to the X-Pack documentation. --- metron-deployment/Kerberos-manual-setup.md | 45 ++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md index 11276dcdc1..3c6e53cd20 100644 --- a/metron-deployment/Kerberos-manual-setup.md +++ b/metron-deployment/Kerberos-manual-setup.md @@ -738,3 +738,48 @@ Here are instructions for enabling X-Pack with Elasticsearch and Kibana: https:/ ``` Once you've performed these steps, you should be able to start seeing data in your ES indexes. + +### X-Pack Common Problems + +#### java.io.FileNotFoundException: File /apps/metron/elasticsearch/xpack-password does not exist + +#### Problem + +The random access indexer topology fails with the following exception. This exception might occur on only some of the Storm worker nodes. + + ``` + 2018-07-02 10:24:08.267 o.a.s.util Thread-8-indexingBolt-executor[3 3] [ERROR] Async loop died! + java.lang.RuntimeException: java.lang.IllegalArgumentException: Unable to read XPack password file from HDFS location '/apps/metron/elasticsearch/xpack-password' + at org.apache.metron.writer.bolt.BulkMessageWriterBolt.prepare(BulkMessageWriterBolt.java:201) ~[stormjar.jar:?] + at org.apache.storm.daemon.executor$fn__10195$fn__10208.invoke(executor.clj:800) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] + at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:482) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] + at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] + at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] + Caused by: java.lang.IllegalArgumentException: Unable to read XPack password file from HDFS location '/apps/metron/elasticsearch/xpack-password' + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.getPasswordFromFile(ElasticsearchUtils.java:201) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.setXPackSecurityOrNone(ElasticsearchUtils.java:187) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.getClient(ElasticsearchUtils.java:147) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.writer.ElasticsearchWriter.init(ElasticsearchWriter.java:53) ~[stormjar.jar:?] + at org.apache.metron.writer.bolt.BulkMessageWriterBolt.prepare(BulkMessageWriterBolt.java:199) ~[stormjar.jar:?] + ... 4 more + Caused by: java.io.FileNotFoundException: File /apps/metron/elasticsearch/xpack-password does not exist + at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606) ~[stormjar.jar:?] + at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819) ~[stormjar.jar:?] + at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596) ~[stormjar.jar:?] + at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) ~[stormjar.jar:?] + at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:140) ~[stormjar.jar:?] + at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341) ~[stormjar.jar:?] + at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) ~[stormjar.jar:?] + at org.apache.metron.common.utils.HDFSUtils.readFile(HDFSUtils.java:55) ~[stormjar.jar:?] + at org.apache.metron.common.utils.HDFSUtils.readFile(HDFSUtils.java:40) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.getPasswordFromFile(ElasticsearchUtils.java:198) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.setXPackSecurityOrNone(ElasticsearchUtils.java:187) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.getClient(ElasticsearchUtils.java:147) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.writer.ElasticsearchWriter.init(ElasticsearchWriter.java:53) ~[stormjar.jar:?] + at org.apache.metron.writer.bolt.BulkMessageWriterBolt.prepare(BulkMessageWriterBolt.java:199) ~[stormjar.jar:?] + ... 4 more + ``` + +#### Solution + +This can occur when an HDFS Client is not installed on the Storm worker nodes. This might occur on any Storm worker node where an HDFS Client is not installed. Installing the HDFS Client on all Storm worker nodes should resolve the problem. From 8776c818ae005c9164e8d4873a58ef44edae0280 Mon Sep 17 00:00:00 2001 From: Nick Allen Date: Tue, 3 Jul 2018 10:17:30 -0400 Subject: [PATCH 3/3] Clarified where the X-Pack plugin needs installed; on each node --- metron-deployment/Kerberos-manual-setup.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md index 3c6e53cd20..e3de148b83 100644 --- a/metron-deployment/Kerberos-manual-setup.md +++ b/metron-deployment/Kerberos-manual-setup.md @@ -538,14 +538,14 @@ In order to correct this, you should: X-Pack ------ -Here are instructions for enabling X-Pack with Elasticsearch and Kibana: https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html - -1. First, stop the random_access_indexing topology through the Storm UI or from the CLI, e.g. +1. Stop the random_access_indexing topology through the Storm UI or from the CLI, e.g. ``` storm kill random_access_indexing ``` +1. [Follow these instructions](https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html) to enable the X-Pack with Elasticsearch and Kibana. The X-Pack plugin will need installed on each of the nodes running Elasticsearch. + 1. You need to be sure to add the appropriate username and password for Elasticsearch and Kibana to enable external connections from Metron components. e.g. the following will create a user "transport_client_user" with password "changeme" and "superuser" credentials. ```