From b20fdb09c51bc26102fb92c147501beff6cd688f Mon Sep 17 00:00:00 2001 From: as22323 Date: Wed, 23 Aug 2017 11:18:07 -0400 Subject: [PATCH 1/7] Update README.md Hello. These updates were based on my experience with using ClearOS/CentOS as a host. List of versions (ClearOS/CentOS environment install) that broke when trying to install. Ansible 2.0.0.2, versions of maven outside of 3.3.9 --- metron-deployment/amazon-ec2/README.md | 47 +++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md index 7b8be93bb4..8e13f255ff 100644 --- a/metron-deployment/amazon-ec2/README.md +++ b/metron-deployment/amazon-ec2/README.md @@ -1,20 +1,42 @@ Apache Metron on Amazon EC2 =========================== -This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron. +This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron. Warning: Amazon will charge for the use of their resources when running Apache Metron. The amount will vary based on the number and size of hosts, along with current Amazon pricing structure. Be sure to stop or terminate all of the hosts instantiated by Apache Metron when not in use to avoid unnecessary charges. Getting Started --------------- +### Downloading Code +Clone or wget the latest Apache Metron release onto a host machine. The host machine will be used to deploy Apache Metron to Amazon EC2. In the instructions below the host machine MacOS. -### Prerequisites +**Example A** (Cloning a branch using git): +``` +git clone -b Metron_ git://git.apache.org/metron.git +``` +*Branch names can be found [here](https://github.com/apache/metron/branches)* -The host used to deploy Apache Metron will need the following software tools installed. The following versions are known to work as of the time of this writing, but by no means are these the only working versions. - - Ansible 2.0.0.2 or 2.2.2.0 +**Example B** (wget): +``` +wget https://github.com/apache/metron/archive/apache-metron-0.4.0-release.tar.gz +tar -xf apache-metron--release.tar.gz +``` +*List of releases can be found [here](https://github.com/apache/metron/releases)* + + +### Prerequisites + +The host used to deploy Apache Metron will need the following software tools installed. The host will also need atleast 6GB of available hard disk space. The following versions are known to work as of the time of this writing. + - Ansible 2.2.2.0 - Python 2.7.11 - - Maven 3.3.9 + - Maven 3.3.9 + +The following componets are also needed. However, the versions listed below are by no means the only working versions. + - Docker 1.12.6 + - nodejs 6.11.1 + - npm 3.10.10 + - bzip2 1.0.6 Any platform that supports these tools is suitable, but the following instructions cover only macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project. @@ -278,3 +300,18 @@ fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => { #### Solution This most often indicates that Ansible cannot connect to the host with the SSH key that it has access to. This could occur if hosts are provisioned with one SSH key, but the playbook is executed subsequently with a different SSH key. The issue can be addressed by either altering the `key_file` variable to point to the key that was used to provision the hosts or by simply terminating all hosts and re-running the playbook. + +Exprimenting with Linux as a Host +--------------- +Update the “ansible.cfg” files below by commenting out the last two lines. +./metron/metron-deployment/amazon-ec2/ansible.cfg +./metron/metron-deployment/ansible.cfg +``` +#[ssh_connection] +#control_path = %(directory)s/%%h-%%p-%%r +``` + +Run the deployment script in the "./metron-deployment/amazon-ec2" folder. +``` +$ ./run.sh +``` From 2e642fd8d1ba01b939b74d597ea5b14c40c228a0 Mon Sep 17 00:00:00 2001 From: as22323 Date: Wed, 23 Aug 2017 11:20:42 -0400 Subject: [PATCH 2/7] Update README.md --- metron-deployment/amazon-ec2/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md index 8e13f255ff..bac635e965 100644 --- a/metron-deployment/amazon-ec2/README.md +++ b/metron-deployment/amazon-ec2/README.md @@ -8,7 +8,7 @@ Warning: Amazon will charge for the use of their resources when running Apache M Getting Started --------------- ### Downloading Code -Clone or wget the latest Apache Metron release onto a host machine. The host machine will be used to deploy Apache Metron to Amazon EC2. In the instructions below the host machine MacOS. +Clone or wget the latest Apache Metron release onto a host machine. The host machine will be used to deploy Apache Metron to Amazon EC2. **Example A** (Cloning a branch using git): ``` From 5f11ca71edc126caa539ac89f568cf9a5c992b5b Mon Sep 17 00:00:00 2001 From: as22323 Date: Wed, 23 Aug 2017 11:22:01 -0400 Subject: [PATCH 3/7] Update README.md --- metron-deployment/amazon-ec2/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md index bac635e965..34331204f5 100644 --- a/metron-deployment/amazon-ec2/README.md +++ b/metron-deployment/amazon-ec2/README.md @@ -311,7 +311,7 @@ Update the “ansible.cfg” files below by commenting out the last two lines. #control_path = %(directory)s/%%h-%%p-%%r ``` -Run the deployment script in the "./metron-deployment/amazon-ec2" folder. +Run the deployment script that is in the "/metron-deployment/amazon-ec2" folder. ``` $ ./run.sh ``` From 21e59b5deb2fbbbd904778b3931acd2f43bea5a0 Mon Sep 17 00:00:00 2001 From: as22323 Date: Wed, 23 Aug 2017 13:48:03 -0400 Subject: [PATCH 4/7] Update README.md --- metron-deployment/amazon-ec2/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md index 34331204f5..311df32d65 100644 --- a/metron-deployment/amazon-ec2/README.md +++ b/metron-deployment/amazon-ec2/README.md @@ -3,7 +3,7 @@ Apache Metron on Amazon EC2 This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron. -Warning: Amazon will charge for the use of their resources when running Apache Metron. The amount will vary based on the number and size of hosts, along with current Amazon pricing structure. Be sure to stop or terminate all of the hosts instantiated by Apache Metron when not in use to avoid unnecessary charges. +Warning: Amazon will charge for the use of their resources when running Apache Metron. The amount will vary based on the number and size of hosts, along with current Amazon pricing structure. Be sure to stop or terminate all hosts instantiated by Apache Metron when not in use to avoid unnecessary charges. Getting Started --------------- @@ -27,18 +27,18 @@ tar -xf apache-metron--release.tar.gz ### Prerequisites -The host used to deploy Apache Metron will need the following software tools installed. The host will also need atleast 6GB of available hard disk space. The following versions are known to work as of the time of this writing. +The host used to deploy Apache Metron will need the following software tools installed. The host will also need at least 6GB of available hard disk space. The following versions are known to work as of the time of this writing. - Ansible 2.2.2.0 - Python 2.7.11 - Maven 3.3.9 -The following componets are also needed. However, the versions listed below are by no means the only working versions. +The following componets are also needed. However, the versions listed below are by no means the only working versions. The following versions are known to work as of the time of this writing. - Docker 1.12.6 - nodejs 6.11.1 - npm 3.10.10 - bzip2 1.0.6 -Any platform that supports these tools is suitable, but the following instructions cover only macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project. +Any platform that supports these tools is suitable, but the following instructions cover using macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project. 1. Install Homebrew by running the following command in a terminal. Refer to the [Homebrew](http://brew.sh/) home page for the latest installation instructions. @@ -301,14 +301,14 @@ fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => { This most often indicates that Ansible cannot connect to the host with the SSH key that it has access to. This could occur if hosts are provisioned with one SSH key, but the playbook is executed subsequently with a different SSH key. The issue can be addressed by either altering the `key_file` variable to point to the key that was used to provision the hosts or by simply terminating all hosts and re-running the playbook. -Exprimenting with Linux as a Host +Experimenting with Linux as Host --------------- Update the “ansible.cfg” files below by commenting out the last two lines. ./metron/metron-deployment/amazon-ec2/ansible.cfg ./metron/metron-deployment/ansible.cfg ``` -#[ssh_connection] -#control_path = %(directory)s/%%h-%%p-%%r +#[ssh_connection] #commented out +#control_path = %(directory)s/%%h-%%p-%%r #commented out ``` Run the deployment script that is in the "/metron-deployment/amazon-ec2" folder. From 357717b7077c43cec233481ba8c875bb5ed52916 Mon Sep 17 00:00:00 2001 From: as22323 Date: Wed, 23 Aug 2017 13:58:53 -0400 Subject: [PATCH 5/7] Update README.md --- metron-deployment/amazon-ec2/README.md | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md index 311df32d65..00a34d7c66 100644 --- a/metron-deployment/amazon-ec2/README.md +++ b/metron-deployment/amazon-ec2/README.md @@ -8,19 +8,20 @@ Warning: Amazon will charge for the use of their resources when running Apache M Getting Started --------------- ### Downloading Code -Clone or wget the latest Apache Metron release onto a host machine. The host machine will be used to deploy Apache Metron to Amazon EC2. + +Use git or wget to download the latest Apache Metron release onto a host machine. The host machine will be used to deploy Apache Metron to Amazon EC2. **Example A** (Cloning a branch using git): ``` -git clone -b Metron_ git://git.apache.org/metron.git +git clone -b Metron_ git://git.apache.org/metron.git ``` *Branch names can be found [here](https://github.com/apache/metron/branches)* **Example B** (wget): ``` -wget https://github.com/apache/metron/archive/apache-metron-0.4.0-release.tar.gz -tar -xf apache-metron--release.tar.gz +wget https://github.com/apache/metron/archive/apache-metron-.tar.gz +tar -xf apache-metron-.tar.gz ``` *List of releases can be found [here](https://github.com/apache/metron/releases)* @@ -303,12 +304,19 @@ This most often indicates that Ansible cannot connect to the host with the SSH k Experimenting with Linux as Host --------------- -Update the “ansible.cfg” files below by commenting out the last two lines. -./metron/metron-deployment/amazon-ec2/ansible.cfg -./metron/metron-deployment/ansible.cfg +Update the “ansible.cfg” files below by commenting out the last two lines.
+
+Files to Update:
+metron/metron-deployment/amazon-ec2/ansible.cfg
+metron/metron-deployment/ansible.cfg
+
+ansible.cfg
``` -#[ssh_connection] #commented out -#control_path = %(directory)s/%%h-%%p-%%r #commented out +. +. +. +#[ssh_connection] #comment +#control_path = %(directory)s/%%h-%%p-%%r #commented ``` Run the deployment script that is in the "/metron-deployment/amazon-ec2" folder. From 526a47a9b499577d8e9eeb9a8d909783ee5fd592 Mon Sep 17 00:00:00 2001 From: as22323 Date: Wed, 23 Aug 2017 14:01:43 -0400 Subject: [PATCH 6/7] Update README.md --- metron-deployment/amazon-ec2/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md index 00a34d7c66..2aec6606b6 100644 --- a/metron-deployment/amazon-ec2/README.md +++ b/metron-deployment/amazon-ec2/README.md @@ -304,9 +304,10 @@ This most often indicates that Ansible cannot connect to the host with the SSH k Experimenting with Linux as Host --------------- +The current ansible.cfg configuration works with macOS. The following changes will allow you to experiment with deploying using Linux. Update the “ansible.cfg” files below by commenting out the last two lines.

-Files to Update:
+Files to update:
metron/metron-deployment/amazon-ec2/ansible.cfg
metron/metron-deployment/ansible.cfg

From 0ef6a7bad3a63e4b3a2a4156a1ae49acbc9f5c88 Mon Sep 17 00:00:00 2001 From: as22323 Date: Wed, 13 Sep 2017 13:54:56 -0400 Subject: [PATCH 7/7] Update ansible.cfg For 0.4.1-rc4, using "control_path = ~/.ssh/ansible-ssh-%%C" for a Linux environment ( such as ClearOS[Distro of CentOS] npm 3.10.10, node 6.11.1, docker 1.12.6, mvn 3.3.9, ansible 2.2.2.0, No Vagrant) could throw the following error: ``` "TASK [setup] ******************************************************************* fatal: [ec2-xx-yy-0-130.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: percent_expand: unknown key %C\r\n", "unreachable": true}" ``` I noticed the line was formally "control_path = %(directory)s/%%h-%%p-%%r" (which worked for the Linux env above) was changed to "control_path = ~/.ssh/ansible-ssh-%%C" This PR is to change the line to "control_path = ~/.ssh/ansbile-ssh-%%h-%%r" to make amazon-ec2 also potentially deployable with Linux. This is an attempt to resolve the path length issue as described in the following links: http://docs.ansible.com/ansible/latest/intro_configuration.html - see control_path https://github.com/ansible/ansible/issues/11536#issuecomment-153030743 - using %%C --- metron-deployment/amazon-ec2/ansible.cfg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/metron-deployment/amazon-ec2/ansible.cfg b/metron-deployment/amazon-ec2/ansible.cfg index 9f3a2fe99d..e22d64caef 100644 --- a/metron-deployment/amazon-ec2/ansible.cfg +++ b/metron-deployment/amazon-ec2/ansible.cfg @@ -24,5 +24,8 @@ forks = 20 log_path = ./ansible.log # fix for "ssh throws 'unix domain socket too long' " problem +#[ssh_connection] +#control_path = ~/.ssh/ansible-ssh-%%C + [ssh_connection] -control_path = ~/.ssh/ansible-ssh-%%C +control_path = ~/.ssh/ansbile-ssh-%%h-%%r