Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

Commit

Permalink
METRON-1239 Drop extra dev environments (nickwallen) closes #852
Browse files Browse the repository at this point in the history
  • Loading branch information
nickwallen committed Nov 30, 2017
1 parent 5a1bb67 commit f084404
Show file tree
Hide file tree
Showing 30 changed files with 47 additions and 1,278 deletions.
20 changes: 10 additions & 10 deletions metron-deployment/Kerberos-manual-setup.md
@@ -1,7 +1,7 @@
Kerberos Setup
==============

This document provides instructions for kerberizing Metron's Vagrant-based development environments; "Quick Dev" and "Full Dev". These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).
This document provides instructions for kerberizing Metron's Vagrant-based development environments. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).

* [Setup](#setup)
* [Setup a KDC](#setup-a-kdc)
Expand All @@ -17,7 +17,7 @@ This document provides instructions for kerberizing Metron's Vagrant-based devel
Setup
-----

1. Deploy a Vagrant development environment; either [Full Dev](vagrant/full-dev-platform/README.md) or [Quick Dev](vagrant/quick-dev-platform/README.md).
1. Deploy the [development environment.](vagrant/full-dev-platform/README.md).

1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant.

Expand Down Expand Up @@ -73,7 +73,7 @@ Setup a KDC
cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
```

1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on [Full Dev](vagrant/full-dev-platform/README.md).
1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on [Full Dev](vagrant/full-dev-platform/README.md).

Edit `/var/kerberos/krb5kdc/kdc.conf` and ensure the following is added to the `realm` section

Expand All @@ -93,9 +93,9 @@ Setup a KDC
/etc/rc.d/init.d/krb5kdc start
chkconfig krb5kdc on
```

1. Start the Kerberos Admin service and ensure that it starts on boot.

```
/etc/rc.d/init.d/kadmin start
chkconfig kadmin on
Expand All @@ -106,9 +106,9 @@ Setup a KDC
```
kadmin.local -q "addprinc admin/admin"
```

1. Setup the `metron` principal. You will `kinit` as the `metron` principal when running topologies. You will be prompted for a password; do not forget it.

```
kadmin.local -q "addprinc metron"
```
Expand All @@ -120,7 +120,7 @@ Verify KDC
1. Ticket renewal is disallowed by default in many Linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies:

```
Exception in thread "main" java.lang.RuntimeException:
Exception in thread "main" java.lang.RuntimeException:
java.lang.RuntimeException: The TGT found is not renewable
```

Expand All @@ -132,8 +132,8 @@ Verify KDC

* If the 'R' flags are present, you may skip to next section.
* If the 'R' flags are absent, you will need to follow the below steps:
1. If the KDC is already setup, then editing `max_life` and `max_renewable_life` in `/var/kerberos/krb5kdc/kdc.conf`, then restarting `kadmin` and `krb5kdc` services will not change the policies for existing users.

1. If the KDC is already setup, then editing `max_life` and `max_renewable_life` in `/var/kerberos/krb5kdc/kdc.conf`, then restarting `kadmin` and `krb5kdc` services will not change the policies for existing users.

You need to set the renew lifetime for existing users and the `krbtgt` realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:

Expand Down
9 changes: 2 additions & 7 deletions metron-deployment/README.md
Expand Up @@ -32,9 +32,7 @@ The Ambari playbook will install a Hadoop cluster including the Metron Services
Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2.

## Vagrant
There are current two Vagrant modes, full-dev and quick-dev. Full-dev installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.
Quick-dev re-installs the core Metron Services (Parsing, Enrichment, and Indexing)on a pre-built instance.
Use quick-dev for testing out changes to core Metron services.
There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.

### Prerequsities
- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+)
Expand All @@ -44,9 +42,6 @@ installed
### Full-Dev
Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`.

### Quick-Dev
Navigate to `metron/metron-deployment/vagrant/quick-dev-platform` and run `vagrant up`.

## Ambari Management Pack
An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.

Expand Down Expand Up @@ -148,7 +143,7 @@ rpm -i <package>

The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.

To create a new version of the file, make any necessary changes to Kibana (e.g. on quick-dev), and export with the appropriate script.
To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev), and export with the appropriate script.

```
python packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboardindex.py \
Expand Down
2 changes: 1 addition & 1 deletion metron-deployment/amazon-ec2/run.sh
Expand Up @@ -62,6 +62,6 @@ $DEPLOYDIR/../scripts/platform-info.sh >> $LOGFILE
cd $DEPLOYDIR
export EC2_INI_PATH=conf/ec2.ini
ansible-playbook -i ec2.py playbook.yml \
--skip-tags="quick_dev,sensor-stubs" \
--skip-tags="sensor-stubs" \
--extra-vars="env=$ENV" \
$EXTRA_ARGS
89 changes: 0 additions & 89 deletions metron-deployment/inventory/quick-dev-platform/group_vars/all

This file was deleted.

47 changes: 0 additions & 47 deletions metron-deployment/inventory/quick-dev-platform/hosts

This file was deleted.

5 changes: 2 additions & 3 deletions metron-deployment/packaging/docker/ansible-docker/README.md
Expand Up @@ -25,16 +25,15 @@ After running the container:
2. run build commands, for example:
- build metron without tests : `mvn clean package -DskipTests`
- build metron and build the rpms as well : `mvn clean install && cd metron-deployment && mvn package -P build-rpms`

If you wish to use this build with a vagrant instance, then after building with rpms as above, modify
your usual vagrant up command to skip the build role, as so:

`vagrant --ansible-skip-tags="build,quick_dev" up`
`vagrant --ansible-skip-tags="build" up`


## Using the Container for deployment

> Note these instructions are outdated
Full instructions are found on the wiki at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361

18 changes: 6 additions & 12 deletions metron-deployment/packaging/packer-build/README.md
Expand Up @@ -6,35 +6,30 @@ Based on the fantastic [Bento](https://github.com/chef/bento) project developed
Images Provided
---------------------
- base-centos-6.7: Centos 6.7 + HDP. Used in the full-dev-platform Vagrant image
- quick-dev-centos-6.7: Centos 6.7 + HDP + Metron. Used for the quick-dev-platform Vagrant image.

Prerequisites
---------------------
- [Packer](https://www.packer.io/) 0.12.2
- [Virtualbox](https://www.virtualbox.org/) 5.0.16+ (Tested with 5.0.20)

Build Both Images
----------------------
----------------------
Navigate to \<your-project-directory\>/metron-deployment/packer-build
Execute bin/bento build

Packer will build both images and export .box files to the ./builds directory.

Build Single Images
----------------------
----------------------
Navigate to *your-project-directory*/metron-deployment/packer-build
* Base Centos (full-dev)
```
bin/bento build base-centos-6.7.json
```
* Quick Dev
```
bin/bento build quick-dev-centos-6.7.json
```

Using Your New Box File
----------------------
Modify the relevant Vagrantfile (full-dev-platform or quick-dev-platform) replacing the lines:
----------------------
Modify the relevant Vagrantfile (full-dev-platform) replacing the lines:
```
<pre><code>config.vm.box = "<i>box_name</i>"
config.ssh.insert_key = true</code></pre>
Expand All @@ -48,4 +43,3 @@ config.ssh.insert_key = true</code></pre>
Launch the image as usual.

Node: Vagrant will cache boxes, you can force Vagrant to reload your box by running <code>vagrant box remove <i>test_box_name</i></code> before launching your new image.

0 comments on commit f084404

Please sign in to comment.