METRON-270: Add Zeppelin to the platform #423

justinleet · 2017-01-25T15:20:10Z

Adds Zeppelin to the Ambari Management Pack portion. Adding to the Ansible / quick dev portion is happening a bit in parallel with Nick's work on METRON-346.

This ticket existed before the mpack, so if we're not comfortable having this just cover the mpack I'd prefer to split this ticket rather than force them together. This isn't packaging up any notebooks, so there's not deviation in functionality until notebooks are added.

Essentially, this ties into metron-indexing (because we rely on data generated there). I'm open to adjusting this if anybody feels strongly about it, but I think it's currently the best place. Zeppelin notebook JSON files are loaded from metron-platform/metron-indexing/src/main/config/zeppelin/. These files can be placed into subdirs for organization if desired. (e.g. zeppelin/bro/bro.json) and must end with .json.

A custom action is added to the mpack to import these notebooks. This action is available regardless of whether or not Metron itself is running. Zeppelin configuration is autopopulated by the management pack.

Zeppelin allows for duplicate notebook names (they'll be given differing IDs). I didn't implement a way to track installed notebooks, but this is potentially a good future feature (to allow us to delete all installed notebooks, etc.). Once the management pack installs the notebooks, they're treated as belonging to Zeppelin entirely and can be managed there.

For testing, I created a few notebooks (valid, invalid, misnamed and in subdirs), updated the spec files, recreated the RPMs, and ran this up on a pretty constrained local cluster. I was able to see them and run sections of them as appropriate.

nickwallen · 2017-01-26T16:57:38Z

So far, I have not been able to get the "Zeppelin Dashboard Install" mechanism to work.

I am not aware of step-by-step instructions to use the MPack, but maybe I am missing them. Thought it would be beneficial to document here since it took me a while to piece all this together. Here is exactly what I did. What did I do wrong?

Build

Build Metron

cd incubator-metron
mvn clean package -DskipTests

Start Docker on your build machine
Build Metron RPMs

mvn clean package -Dbuild-rpms -DskipTests

Validate Metron RPMs

$ find ./ -name "*.rpm"
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-common-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-data-management-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-elasticsearch-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-enrichment-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-indexing-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-parsers-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-pcap-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-profiler-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-solr-0.3.0-201701261504.noarch.rpm
.//metron-deployment/packaging/docker/rpm-docker/SRPMS/metron-0.3.0-201701261504.src.rpm

Build Ambari MPack

cd metron-deployment
mvn clean package

Validate Ambari MPack

$ find . -name "*mpack*.tar.gz"
./packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz

Create Test VM

Launch Vagrant and install Ambari only

cd metron-deployment/vagrant/quick-dev
vagrant --ansible-tags=ambari up

Copy Artifacts to VM

scp incubator-metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm vagrant@node1:/tmp
scp incubator-metron/metron-deploment/packaging/ambari/metron-mpack/target/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz vagrant@node1:/tmp

Prepare RPMS

vagrant ssh
sudo su -
mkdir /localrepo
cp /tmp/metron*.rpm /localrepo

Install Ambari MPack

Install MPack

ambari-server install-mpack --mpack=/tmp/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
service ambari-server restart

Deploy Metron with MPack

You may need to clear the browser cache to see the additional options installed by the MPack.
Login to Ambari at http://node1:8080
Click Add Services
Select "Metron" and then continue the install
SUCCESS! Validated that I was prompted to install Zeppelin and Spark as dependencies

Install Zeppelin Notebooks

Create a Zeppelin notebook and drop it at /usr/metron/0.3.0/zeppelin/.
In Ambari, Click Metron > Service Actions > Zeppelin Dashboard Install.
This is what I see from the Ambari status window.

2017-01-26 16:45:29,926 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.5.0.0-1245
2017-01-26 16:45:29,926 - Checking if need to create versioned conf dir /etc/hadoop/2.5.0.0-1245/0
2017-01-26 16:45:29,927 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-01-26 16:45:29,947 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 exist already', '')
2017-01-26 16:45:29,948 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-01-26 16:45:29,968 - checked_call returned (0, '')
2017-01-26 16:45:29,972 - Ensuring that hadoop has the correct symlink structure
2017-01-26 16:45:29,972 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf

Command completed successfully!

FAILURE! I then go to Zeppelin at http://node1:9995/ and search for the notebook. The Notebook does not seem to be installed.

nickwallen

A few comments. Thanks for the PR.

nickwallen · 2017-01-26T17:01:49Z

...n-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py

+        from params import params
+        env.set_params(params)
+
+        for dirName, subdirList, files in os.walk(params.metron_config_zeppelin_path):


Would it make sense to log the path that it is looking for the Notebooks in; params.metron_config_zeppelin_path? I'd like to see this in the Ambari status output, especially if it doesn't find any Notebooks.

I'll add logging for that. You're right, in the case that there are no notebooks, it'd be helpful to see what it's using.

nickwallen · 2017-01-26T17:02:39Z

...packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml

@@ -187,6 +187,14 @@
                  <timeout>600</timeout>
                </commandScript>
              </customCommand>
+              <customCommand>
+                <name>ZEPPELIN_DASHBOARD_INSTALL</name>


Should we call this "Zeppelin Notebook Install"? I think that's the preferred Zeppelin nomenclature.

Seems like they use "Import" instead of "Install". How about "Zeppelin Notebook Import"?

nickwallen · 2017-01-26T18:15:17Z

I had the wrong directory. If I put a Notebook in /usr/metron/0.3.0/config/zeppelin/ then it works properly. Nice!

2017-01-26 18:12:53,238 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.5.0.0-1245
2017-01-26 18:12:53,238 - Checking if need to create versioned conf dir /etc/hadoop/2.5.0.0-1245/0
2017-01-26 18:12:53,240 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1}
2017-01-26 18:12:53,273 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 exist already', '')
2017-01-26 18:12:53,273 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False}
2017-01-26 18:12:53,295 - checked_call returned (0, '')
2017-01-26 18:12:53,296 - Ensuring that hadoop has the correct symlink structure
2017-01-26 18:12:53,296 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-01-26 18:12:53,299 - Execute['curl -s -XPOST http://node1:9995/api/notebook/import -d "@/usr/metron/0.3.0/config/zeppelin/Metron - YAF Telemetry.json"'] {'logoutput': True}
{"status":"CREATED","message":"","body":"2C88YEF19"}

Command completed successfully!

nickwallen · 2017-01-26T19:06:40Z

metron-deployment/README.md

@@ -133,6 +133,8 @@ Notably, the URL for the GeoIP database that is preloaded (and is prefilled by d

 After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearch templates.  Similar to this, a custom Kibana action to Load Template is available.

+Another custom action is available in Ambari to import Zeppelin dashboards.


Should we link from this mention here to the more detailed information that you have in metron-platform/metron-indexing/README.md?

mattf-apache · 2017-01-26T20:28:47Z

@nickwallen , glad you got it working. Just BTW, the wiki page recently posted by Dima Kovalyov ( @DimDroll ) has fairly detailed step-by-step instructions for Ambari install, which it looks like you pretty well replicated. It will need updating after @dlyle65535 commits the stuff from METRON-609 and METRON-634 (see PR#425).

Would be great if you or @justinleet add the Zeppelin steps to that wiki page :-)

nickwallen

+1 Works great. Tested by manually pushing a Notebook to the local filesystem. Also, tested by packaging a notebook in the RPM.

justinleet added 4 commits January 23, 2017 08:36

Working initially

e5b392c

Building directory for loading notebook files

1cafd1d

Updating READMEs

58f2ed3

Removing extraneous spec change from testing, and updating README

c7e3958

nickwallen reviewed Jan 26, 2017

View reviewed changes

justinleet added 2 commits January 26, 2017 12:27

review changes

eb6b0ae

Updating README

407382b

nickwallen reviewed Jan 26, 2017

View reviewed changes

Updating deployment README.md to link indexing README

f7fc800

nickwallen approved these changes Jan 26, 2017

View reviewed changes

nickwallen mentioned this pull request Jan 26, 2017

METRON-676 Create Zeppelin Notebook for YAF Telemetry #427

Closed

asfgit closed this in 9890018 Jan 27, 2017

justinleet deleted the zeppelin_dashboard branch April 4, 2017 12:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

METRON-270: Add Zeppelin to the platform #423

METRON-270: Add Zeppelin to the platform #423

justinleet commented Jan 25, 2017

nickwallen commented Jan 26, 2017

nickwallen left a comment

nickwallen Jan 26, 2017

justinleet Jan 26, 2017

nickwallen Jan 26, 2017

justinleet Jan 26, 2017

nickwallen Jan 26, 2017

nickwallen commented Jan 26, 2017

nickwallen Jan 26, 2017

mattf-apache commented Jan 26, 2017 •

edited

nickwallen left a comment

		@@ -133,6 +133,8 @@ Notably, the URL for the GeoIP database that is preloaded (and is prefilled by d

		After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearch templates. Similar to this, a custom Kibana action to Load Template is available.

		Another custom action is available in Ambari to import Zeppelin dashboards.

METRON-270: Add Zeppelin to the platform #423

METRON-270: Add Zeppelin to the platform #423

Conversation

justinleet commented Jan 25, 2017

nickwallen commented Jan 26, 2017

Build

Create Test VM

Install Ambari MPack

Deploy Metron with MPack

Install Zeppelin Notebooks

nickwallen left a comment

Choose a reason for hiding this comment

nickwallen Jan 26, 2017

Choose a reason for hiding this comment

justinleet Jan 26, 2017

Choose a reason for hiding this comment

nickwallen Jan 26, 2017

Choose a reason for hiding this comment

justinleet Jan 26, 2017

Choose a reason for hiding this comment

nickwallen Jan 26, 2017

Choose a reason for hiding this comment

nickwallen commented Jan 26, 2017

nickwallen Jan 26, 2017

Choose a reason for hiding this comment

mattf-apache commented Jan 26, 2017 • edited

nickwallen left a comment

Choose a reason for hiding this comment

mattf-apache commented Jan 26, 2017 •

edited