Updated readme with more info about post-install script issues

README.md

@@ -1,13 +1,13 @@
 <img src="https://cloud.githubusercontent.com/assets/22897558/22356169/8f6e0b16-e3ec-11e6-8695-2273424c2b06.png" width="200" />
 
-# Overview 
+# Overview
 
 PcapDB is a distributed, search-optimized open source packet capture system. It was designed to
-replace expensive, commercial appliances with off-the-shelf hardware and a free, easy to manage 
+replace expensive, commercial appliances with off-the-shelf hardware and a free, easy to manage
 software system. Captured packets are reorganized during capture by flow (an indefinite length
 sequence of packets with the same src/dst ips/ports and transport proto), indexed by flow, and
 searched (again) by flow. The indexes for the captured packets are relatively tiny (typically less
-than 1% the size of the captured data). 
+than 1% the size of the captured data).
 
 For hardware requirements, see [HARDWARE.md](HARDWARE.md).
 
@@ -16,13 +16,13 @@ For hardware requirements, see [HARDWARE.md](HARDWARE.md).
 ## DESTDIR
 Many things in this file refer to DESTDIR as a pathname prefix. The default, and that used by future pcapdb packages, is `/var/pcapdb`.
 
-## Architectural Overview 
+## Architectural Overview
 A PcapDB installation consists of a Search Head and one or more Capture Nodes. The Search Head can
 also be a Capture Node, or it can be a VM somewhere else. Wherever it is, the Search Head must be
-accessible by the Capture Nodes, but there's no need for the Capture Nodes to be visible to the 
+accessible by the Capture Nodes, but there's no need for the Capture Nodes to be visible to the
 Search Head.
 
-# 1. Requirements 
+# 1. Requirements
 PcapDB is designed to work on Linux servers only. It was developed on both Redhat Enterprise and
 Debian systems, but its primary testbed has so far been Redhat based. While it has been verified to
 work (with packages from non-default repositories) on RHEL 6, a more bleeding edge system (like
@@ -33,7 +33,7 @@ pcapdb. They are easiest to install on modern Debian based machines.
 
 requirements.txt contains python/pip requirements. They will automatically be installed via 'make install'.
 
-# 2. Installing 
+# 2. Installing
 To build and install everything in /var/pcapdb/, run one of:
 ```
 make install-search-head
@@ -42,7 +42,7 @@ make install-monolithic
 ```
  - Like with most Makefiles, you can set the DESTDIR environment variable to specify where to
    install the system. `make install-search-head DESTDIR=/var/mypcaplocation`
- - This includes installing in place: `make install-capture-node DESTDIR=$(pwd)`. In this case, PcapDB 
+ - This includes installing in place: `make install-capture-node DESTDIR=$(pwd)`. In this case, PcapDB
    won't install system scripts for various needed components. You will have to run it manually, see
    below.
  - If you're behind a proxy, you'll need to specify a proxy connection string using PROXY=host:port as
@@ -54,23 +54,23 @@ Postgresql may install in a strange location, as noted in the 'indexer/README'.
 failures in certain pip installed packages. Add `PATH=$PATH:<pgsql_bin_path>` to the end of your
 'make install' command to fix this. For me, it is: `make install PATH=$PATH:/usr/pgsql-9.4/bin`.
 
-# 3. Setup 
-After running 'make install', there are a few more steps to perform. 
+# 3. Setup
+After running 'make install', there are a few more steps to perform.
 
 ## 3-1: Post-Install script
 The core/bin/post-install.sh script will handle the vast majority of the system setup for you.
- - It does so idempotently, so it can be run multiple times without breaking anything.
+ - Only run once. It can cause issues if you run it multiple times. ~~It does so idempotently, so it can be run multiple times without breaking anything.~~ If you run it again, you may have to re-install completely. Moving the old PcapDB install files to another directory can cause problems as well.
  - Run without arguments to get the usage information.
-  - Basically, you want to give it arguments based on whether you're setting up a search head (-s), 
-    a capture node (-c), or monolithic install (-c -s). 
+  - Basically, you want to give it arguments based on whether you're setting up a search head (-s),
+    a capture node (-c), or monolithic install (-c -s).
   - You'll also have to give it the search head's IP.
 ```
 /var/pcapdb/core/bin/post-install.sh [-c] [-s] <search_head_ip>    
 ```
 
 This will set up the databases and rabbitmq.
 
-## 3-2 DESTDIR/etc/pcapdb.cfg 
+## 3-2 DESTDIR/etc/pcapdb.cfg
 This is the main Pcapdb config file. You must set certain values before PcapDB will run at all.
 There are a few things you need to set in here manually:
  - __(On capture nodes) The search head db password__
@@ -86,18 +86,18 @@ You'll need to create an admin user.
 sudo su - capture
 ./bin/python core/manage.py add_user <username> <first_name> <last_name> <email>
 ```
- - Usernames must be at least four characters long.`
+ - Usernames must be at least four characters long.
  - This will email you a link to use to set that user's password.
   - (This is why email had to be set up).
   - root@localhost is a reasonable email address, if you need it.
   - *Note that manage.py also has a __createsuperuser__ command, which shouldn't be used.*
 
-## 3-4 That should be it. 
+## 3-4 That should be it.
 You should be able to login with your admin account.
 
 ### Things that can, and have, gone wrong
  - If your host doesn't have a host name in DNS, you can set an IP in the 'search\_head\_host' variable
-   in the pcapdb.cfg file. 
+   in the pcapdb.cfg file.
 
 ## 3-5 pfring-zc drivers
 One more thing. You should install the drivers specific to your capture card for pfring-zc. The
@@ -115,19 +115,19 @@ Every capture node belongs to a capture site.
  - These can be LDAP groups.
 
 ## Create a Capture Node
-Add a capture node to your site. 
+Add a capture node to your site.
  - If the search head is also a capture node, it will have to be added to.
  - If the buttons for disk and capture configuration aren't active, your user isn't an admin for the relevant site.
 
 ## Setup disks
 PcapDB needs three types of disk:
- - An OS disk, which is barely used. 
+ - An OS disk, which is barely used.
  - An index disk or disks. It's generally recommended to have two identical disks for this to set up a RAID 1.
- - A bunch of disk for capture storage. 
+ - A bunch of disk for capture storage.
 
- You've obviously already got an OS at this point. The other two can be any sort of hard disk, SSD, partition, or RAID. As long as the system thinks it's a block device, you should be fine. 
+ You've obviously already got an OS at this point. The other two can be any sort of hard disk, SSD, partition, or RAID. As long as the system thinks it's a block device, you should be fine.
 
-#### Note 
+#### Note
 *The system is pretty dumb about what disks it considers to be block devices. Any /dev/sd or /dev/md device works, for now*
 
 ### Add an Index Disk
@@ -137,25 +137,25 @@ In the disk management interface, select one or two identical disks to act as th
 ### Add a Capture Disk
 In the disk management interface, you can build RAID 5 arrays, and then assign those (or individual disks) as capture disks.
  - While you can add all your disks individually as capture disks, it's recommended to RAID them for a bit of data safety.  RAID's of 9 disks are fairly reasonable.
- - Like with the index disk, select groups of disks, and click 'Create RAID 5'. Once that's done, you can add the 
+ - Like with the index disk, select groups of disks, and click 'Create RAID 5'. Once that's done, you can add the
    resulting disk as a capture disk.
  - Or you can add individual disks (or external RAIDs) as capture disks.
  - Once a capture disk is added, it must be activated above before it can be captured to.
  - You can also set disks as __spares__, which are shared across all RAIDS created by PcapDB.
  - If RAIDS are ever degraded, they should be put in REPAIR mode automatically. They'll still be available to search, but won't be written to until they're fixed.
 
 ## Set a capture interface, and go.
-In the capture interface, enable the interface or interfaces of your choice. 
+In the capture interface, enable the interface or interfaces of your choice.
  - Each will get a separate thread (which will in turn be dedicated to it's own processor. So you shouldn't try to capture on more interfaces than half your CPU's).
- - PFring mode is far less likely to drop packets than libpcap mode. 
+ - PFring mode is far less likely to drop packets than libpcap mode.
  - PFringZC mode is far less likely to drop packets than PFring mode, but requires a license from NTOP.
 
 # Details on the various subsystems
 PcapDB uses quite a few off-the-shelf open source systems, and it's useful to understand how those
 pieces fit into the larger system. What follows is a detailed description of those systems, and how
-to set them up manually. 
+to set them up manually.
 
-## RabbitMQ 
+## RabbitMQ
 RabbitMQ is a fast and efficient messaging system used to communicate simple messages between a
 distributed network of hosts. As with Celery, RabbitMQ is really meant for distributing messages to
 the 'first available' worker, but in PcapDB all of our messages are to a specific worker. As such,
@@ -174,26 +174,26 @@ rabbitmqctl add_user pcapdb <a strong password>
 rabbitmqctl set_permissions -p / pcapdb '.*' '.*' '.*'
 ```
 
-## Database Setup 
-The setup varies significantly between the search head and capture nodes. 
+## Database Setup
+The setup varies significantly between the search head and capture nodes.
 
-### Add the 'capture' Role 
+### Add the 'capture' Role
 On all pcapdb servers, add a 'capture' role with login privileges and a password. As the postgres user:
 ```
 sudo su -postgres
 createuser capture -l -P
 ```
 
 The 'db\_pass' variable in the pcapdb.cfg file should be set to the Search Head's db password on all
-pcapdb hosts in the network. 
+pcapdb hosts in the network.
 
-### On the Search Head 
+### On the Search Head
 Create a database named "pcapdb":
 ```
 createdb -O capture pcapdb
 ```
 
-Edit the Search Head's "pg\_hba.conf" (location varies) file to allow connections to Search Head DB from localhost. Also add a line allowing each capture node. 
+Edit the Search Head's "pg\_hba.conf" (location varies) file to allow connections to Search Head DB from localhost. Also add a line allowing each capture node.
 ```
 host    pcapdb          capture         127.0.0.1/32            md5
 host    pcapdb          capture         <capture node ip>       md5
@@ -204,7 +204,7 @@ Edit the Search Head's postgresql.conf file so that it listens on it's own IP:
 listen_addresses = 'localhost,<search head ip>'
 ```
 
-### On the Capture Nodes 
+### On the Capture Nodes
 Create a database named 'capture\_node' on each capture node host:
 ```
 createdb -O capture capture_node
@@ -220,8 +220,8 @@ should be needed.
 service postgresql restart
 ```
 
-### Install the Database Tables 
-After restarting the postgres service, we'll need to install the database tables on each 
+### Install the Database Tables
+After restarting the postgres service, we'll need to install the database tables on each
 PcapDB host. From the PcapDB installation directory (typically /var/pcapdb):
 ```
 sudo su - root
@@ -236,66 +236,66 @@ login_gui search_head_gui capture_node_api
 ./bin/python core/manage.py migrate --database=capture_node
 ```
 
-## Web server setup 
+## Web server setup
 The Makefile will generate a self-signed cert for your server for you if an installed one doesn't
 already exist at '/etc/ssl/<HOSTNAME>.pem'. You should probably change that to something that isn't
 self-signed.
 
-## Firewall Notes 
+## Firewall Notes
  - The Capture Nodes don't open any incoming ports for PcapDB, all communication is out to the Search Head.
  - Using IPtables or other system firewalls on the Capture Nodes is discouraged. Instead
    put them on a normally inaccessible network. They shouldn't generally have any ports open other than ssh.
  - The Search Head needs to be accessible by the Capture Nodes on ports 443, 5432 (postgres), and
-   25672 (rabbitmq). 
+   25672 (rabbitmq).
    - It's ok to us IP tables on the Search Head (and will eventually be automatic)
 
-## Running the system 
+## Running the system
 If you installed anywhere except 'in place', the system should attempt to run itself via
 supervisord. __You'll have to restart some processes, as supervisord will have given up on them.__
  - The `supervisorctl` command can give you the status of the various components of the system. Capture has to be started manually from within the interface, so you shouldn't expect it to be running initially.
- - The `capture_runner` process, however, should be running. From within supervisorctl, 
+ - The `capture_runner` process, however, should be running. From within supervisorctl,
 
  - The `core/runserver` and `core/runcelery` scripts will be helpful when not running the system in
    production.
- - Similarly, to run capture outside of production, use the capture_runner.py script: 
+ - Similarly, to run capture outside of production, use the capture_runner.py script:
    `DESTDIR/bin/python DESTDIR/core/bin/capture_runner.py`
    - DESTDIR/log/django.log will tell you the exact command used to start capture, if for some reason it's failing to start.
    - DESTDIR/log/capture.log will usually give you some idea why capture is failing to run. If this file doesn't exist, either capture has never successfully ran at all, or rsyslog isn't forwarding the logs to the right place.
 
 
-### PostgreSQL 
+### PostgreSQL
 While PcapDB is a database of packets, it uses postgres to take care of more mundane database tasks.
 The Search Head has a unique database that houses information on users, capture nodes, celery
 response data, and aggregate statistics for the entire network of PcapDB capture nodes. All PcapDB
-hosts must be able to connect to the Search Head database. 
+hosts must be able to connect to the Search Head database.
 
 Each Capture Node also has a database that keeps track of the available disk chunks and indexes on
 that node. This database is only accessible to the capture node itself.
 
 This has to be set up manually. See above for more information.
 
-### Celery 
+### Celery
 Celery is a system for distributing and scheduling tasks across a network of workers. PcapDB manages
 all of it's communications with the Capture Nodes through Celery tasks, from initiating searches to
 managing disk arrays. The tasks are assigned and picked up by the appropriate host via RabbitMQ
 messaging queues, and the responses are saved to the search head via the search head's database.
 Celery runs on both the Search Head and all Capture Nodes, though each host subscribes to different
-task queues (see RabbitMQ below). 
+task queues (see RabbitMQ below).
 
 Celery is configured automatically on system install, and the process is managed via supervisord.
 
 
-### uWSGI and Nginx 
+### uWSGI and Nginx
 The web interface for PcapDB is built in the Python Django system, which is served via a unix
 socket using uWSGI and persistant Python instances. Nginx handles all of the standard HTTP/HTTPS
 portions of the web service, and passes Django requests to uWSGI via it's socket. (This is a pretty
-standard way of doing things). 
+standard way of doing things).
 
  - uWSGI and Nginx are automatically configured on install.
  - Nginx is managed as a standard system service.
  - uWSGI is managed via supervisord
 
-#### Certificates 
+#### Certificates
 The standard configuration for PcapDB and Nginx expects ssl certificates and a private key installed
 at:
 ```
@@ -310,15 +310,15 @@ automatically (with your input).
 # A Few Other Tasks
 
 ## Install static files
-You will need to install the static files for pcapdb. 
+You will need to install the static files for pcapdb.
 
 ```
 sudo su - capture
 ./bin/python core/manage.py collectstatic
 ```
-# Making sure everything is working. 
+# Making sure everything is working.
 After installing and setting up the database, there are a few things you can check to make sure
-everything is working. 
+everything is working.
 
  1. Restart supervisord to reset the uwsgi and celery processes to pick up the new database configs.