Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

METRON-1239 Drop extra dev environments #852

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
20 changes: 10 additions & 10 deletions metron-deployment/Kerberos-manual-setup.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Kerberos Setup
==============

This document provides instructions for kerberizing Metron's Vagrant-based development environments; "Quick Dev" and "Full Dev". These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).
This document provides instructions for kerberizing Metron's Vagrant-based development environments. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).

* [Setup](#setup)
* [Setup a KDC](#setup-a-kdc)
Expand All @@ -17,7 +17,7 @@ This document provides instructions for kerberizing Metron's Vagrant-based devel
Setup
-----

1. Deploy a Vagrant development environment; either [Full Dev](vagrant/full-dev-platform/README.md) or [Quick Dev](vagrant/quick-dev-platform/README.md).
1. Deploy the [development environment.](vagrant/full-dev-platform/README.md).

1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant.

Expand Down Expand Up @@ -73,7 +73,7 @@ Setup a KDC
cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
```

1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on [Full Dev](vagrant/full-dev-platform/README.md).
1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on [Full Dev](vagrant/full-dev-platform/README.md).

Edit `/var/kerberos/krb5kdc/kdc.conf` and ensure the following is added to the `realm` section

Expand All @@ -93,9 +93,9 @@ Setup a KDC
/etc/rc.d/init.d/krb5kdc start
chkconfig krb5kdc on
```

1. Start the Kerberos Admin service and ensure that it starts on boot.

```
/etc/rc.d/init.d/kadmin start
chkconfig kadmin on
Expand All @@ -106,9 +106,9 @@ Setup a KDC
```
kadmin.local -q "addprinc admin/admin"
```

1. Setup the `metron` principal. You will `kinit` as the `metron` principal when running topologies. You will be prompted for a password; do not forget it.

```
kadmin.local -q "addprinc metron"
```
Expand All @@ -120,7 +120,7 @@ Verify KDC
1. Ticket renewal is disallowed by default in many Linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies:

```
Exception in thread "main" java.lang.RuntimeException:
Exception in thread "main" java.lang.RuntimeException:
java.lang.RuntimeException: The TGT found is not renewable
```

Expand All @@ -132,8 +132,8 @@ Verify KDC

* If the 'R' flags are present, you may skip to next section.
* If the 'R' flags are absent, you will need to follow the below steps:
1. If the KDC is already setup, then editing `max_life` and `max_renewable_life` in `/var/kerberos/krb5kdc/kdc.conf`, then restarting `kadmin` and `krb5kdc` services will not change the policies for existing users.

1. If the KDC is already setup, then editing `max_life` and `max_renewable_life` in `/var/kerberos/krb5kdc/kdc.conf`, then restarting `kadmin` and `krb5kdc` services will not change the policies for existing users.

You need to set the renew lifetime for existing users and the `krbtgt` realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:

Expand Down
9 changes: 2 additions & 7 deletions metron-deployment/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,7 @@ The Ambari playbook will install a Hadoop cluster including the Metron Services
Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2.

## Vagrant
There are current two Vagrant modes, full-dev and quick-dev. Full-dev installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.
Quick-dev re-installs the core Metron Services (Parsing, Enrichment, and Indexing)on a pre-built instance.
Use quick-dev for testing out changes to core Metron services.
There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.

### Prerequsities
- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+)
Expand All @@ -44,9 +42,6 @@ installed
### Full-Dev
Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`.

### Quick-Dev
Navigate to `metron/metron-deployment/vagrant/quick-dev-platform` and run `vagrant up`.

## Ambari Management Pack
An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.

Expand Down Expand Up @@ -148,7 +143,7 @@ rpm -i <package>

The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.

To create a new version of the file, make any necessary changes to Kibana (e.g. on quick-dev), and export with the appropriate script.
To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev), and export with the appropriate script.

```
python packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboardindex.py \
Expand Down
2 changes: 1 addition & 1 deletion metron-deployment/amazon-ec2/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,6 @@ $DEPLOYDIR/../scripts/platform-info.sh >> $LOGFILE
cd $DEPLOYDIR
export EC2_INI_PATH=conf/ec2.ini
ansible-playbook -i ec2.py playbook.yml \
--skip-tags="quick_dev,sensor-stubs" \
--skip-tags="sensor-stubs" \
--extra-vars="env=$ENV" \
$EXTRA_ARGS
89 changes: 0 additions & 89 deletions metron-deployment/inventory/quick-dev-platform/group_vars/all

This file was deleted.

47 changes: 0 additions & 47 deletions metron-deployment/inventory/quick-dev-platform/hosts

This file was deleted.

5 changes: 2 additions & 3 deletions metron-deployment/packaging/docker/ansible-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,15 @@ After running the container:
2. run build commands, for example:
- build metron without tests : `mvn clean package -DskipTests`
- build metron and build the rpms as well : `mvn clean install && cd metron-deployment && mvn package -P build-rpms`

If you wish to use this build with a vagrant instance, then after building with rpms as above, modify
your usual vagrant up command to skip the build role, as so:

`vagrant --ansible-skip-tags="build,quick_dev" up`
`vagrant --ansible-skip-tags="build" up`


## Using the Container for deployment

> Note these instructions are outdated

Full instructions are found on the wiki at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361

18 changes: 6 additions & 12 deletions metron-deployment/packaging/packer-build/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,35 +6,30 @@ Based on the fantastic [Bento](https://github.com/chef/bento) project developed
Images Provided
---------------------
- base-centos-6.7: Centos 6.7 + HDP. Used in the full-dev-platform Vagrant image
- quick-dev-centos-6.7: Centos 6.7 + HDP + Metron. Used for the quick-dev-platform Vagrant image.

Prerequisites
---------------------
- [Packer](https://www.packer.io/) 0.12.2
- [Virtualbox](https://www.virtualbox.org/) 5.0.16+ (Tested with 5.0.20)

Build Both Images
----------------------
----------------------
Navigate to \<your-project-directory\>/metron-deployment/packer-build
Execute bin/bento build

Packer will build both images and export .box files to the ./builds directory.

Build Single Images
----------------------
----------------------
Navigate to *your-project-directory*/metron-deployment/packer-build
* Base Centos (full-dev)
```
bin/bento build base-centos-6.7.json
```
* Quick Dev
```
bin/bento build quick-dev-centos-6.7.json
```

Using Your New Box File
----------------------
Modify the relevant Vagrantfile (full-dev-platform or quick-dev-platform) replacing the lines:
----------------------
Modify the relevant Vagrantfile (full-dev-platform) replacing the lines:
```
<pre><code>config.vm.box = "<i>box_name</i>"
config.ssh.insert_key = true</code></pre>
Expand All @@ -48,4 +43,3 @@ config.ssh.insert_key = true</code></pre>
Launch the image as usual.

Node: Vagrant will cache boxes, you can force Vagrant to reload your box by running <code>vagrant box remove <i>test_box_name</i></code> before launching your new image.