Skip to content

Multiple Host HOWTO

Jump to bottom Edit New page
Maddalena, Art edited this page Sep 23, 2016 · 21 revisions

Going from a single host to a multiple host deployment isn't too difficult. The biggest issues are around opening up elasticsearch to more than just localhost and getting the elasticsearch configuration right. These instructions assume you've used the easybutton scripts to create a single machine moloch cluster and installed everything into /data/moloch.

Moving ElasticSearch

If you still have the single node elasticsearch you can either leave it in place and add more elasticsearch nodes to the cluster OR move it to a new machine first. If you choose to move it here are some basic instructions for moving, otherwise you can skip this step. For big elasticsearch clusters it is recommended that you use REAL Java 7.

  • Shutdown the cluster
curl -XPOST http://localhost:9200/_shutdown
  • Confirm the node has shutdown
ps aux | grep elastic
  • tar up everything
tar cvzf /tmp/moloch_all.tgz /data/moloch
  • Copy tar file to new machine and untar. NOTE: Only copy the /data/moloch/data directory to ONE new machine, not all the new machines.
scp /tmp/moloch_all.tgz newmachine:/tmp

Give ES enough memory

Deciding how much memory to give ES per machine environment can be tricky, especially in a multi ES node. ES articles suggest anywhere between 1/3 and 1/2 of memory should be reserved for disk cache.

On the NEW machine(s) edit /data/moloch/bin/run_es.sh and change the ES_HEAP_SIZE to about 2/3 the amount of the free memory you have, up to 30G. This leaves 1/3 for the disk cache. In general you don't want to give ES more then 30G so that it can use Compressed OOPs.

For large memory machines it is possible to run multiple nodes per machine. Usually you want to run a few large nodes instead of multiple small nodes because of ES overhead. The only exception to this is in research instances (where you never delete data) and you aren't sure how big a cluster you need. Running smaller instances allows you to move them later if you grow. Our current recommendations - 32G of memory run 1 20G node, 48G of memory run 1 20G node, 64G of memory run 1 30G nodes, 96G of memory run 2 22G nodes, 128G of memory run 2 30G nodes. Use Oracle Java 7 for big clusters, and uncomment the export JAVA_OPTS="-XX:+UseCompressedOops" line in the run_es.sh file. (Java 7 enables this by default now, so this option is probably no longer required.)

For more information and why 30G is the max recommended setting read the heap sizing guide

Open up ElasticSearch

The easybutton scripts by default locks down elasticsearch to the local machine. If there will be multiple machines in the elasticsearch cluster you will need modify your central ElasticSearch cluster (single node at this point) config so that it listens to your local network. Security of your ES cluster is your responsibility! If you expose ElasticSearch to the internet at large, bad things are easy to accomplish.

  • Edit '/data/moloch/etc/elasticsearch.yml'
  • Change network.bind_host from 127.0.0.1 to 0.0.0.0
  • Change discovery.zen.ping.unicast.hosts from localhost to the fqdn
  • Shutdown your ElasticSearch cluster (execute on your central ES node).
curl -XPOST http://localhost:9200/_shutdown
  • Start ElasticSearch
/data/moloch/bin/run_es.sh
  • Modify the Moloch capture config file /data/moloch/etc/config.ini so that it communicates with your central elasticsearch tier.

Change elasticsearch=localhost:9200

to elasticsearch=${YOUR_CENTRAL_ES_LISTENER_IP}:9200

  • Confirm the single sniffer node is still operational before bringing up a new sniffer node.

Adding Elastic Search Nodes

If you've already followed the opening up instructions you don't need to shutdown and restart the node you already have. Note: It may save debugging time if you do. Make sure you have the same yml file everywhere.

NOTE: Today and past days will still use the same number of shards as previously. Only FUTURE days (sessions-* indexes) will have more shards since by default Moloch uses the number of shards equal to the number of nodes.

  • Edit elasticsearch.yml file - ALL ES nodes should have the SAME yml file, unless you are running a separate no data master.
  • Change network.bind_host from 127.0.0.1 to 0.0.0.0
  • Change gateway.recover_after_nodes and gateway.expected_nodes to the TOTAL number of ES nodes
  • Update discovery.zen.ping.unicast.hosts with up to 4 ES node names.
  • Copy the elasticsearch.yml file to all ES nodes
  • Read section above about Give ES enough memory
  • While cluster is still running run the following to prevent shard shuffling when adding nodes until we are ready
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable_allocation": true}}'
  • OPTIONAL: Shutdown current nodes (REQUIRED if ES was not previously opened up, otherwise you can skip)
curl -XPOST http://localhost:9200/_shutdown
  • Verify that the limits are set correctly on the new machines by checking ulimit -a. Open files should be at least 65k, max locked memory should be at least whatever size you've given ES, and max user processes should be at least 10240. Most of these are set in /etc/security/limits.conf however on Centos 6 max user processes is set in /etc/security/limits.d/90-nproc.conf
  • Start all nodes (or just the new ones if you didn't issue the shutdown), ES can be flaky if starting up the entire cluster at once, it is usually best to start a single node (one of the nodes listed in discovery.zen.ping.unicast.hosts) and wait 10 seconds before starting the rest of the nodes.
/data/moloch/bin/run_es.sh
  • Verify the new elasticsearch node(s) show up in head/bigdesk, if they don't show up check the logs (/data/moloch/logs/Moloch.log by default) and firewall issues.
http://moloches.fqdn:9200/_plugin/head/
http://moloches.fqdn:9200/_plugin/bigdesk/
  • Even with all the config, ES sometimes goes all split brain. If you never go YELLOW/GREEN, check head/bigdesk on each of the nodes. If you see a split brain situation, restart just that node.
curl -XPOST 'http://bad-moloch-node.fqdn:9200/_cluster/nodes/_local/_shutdown'
  • Upgrade DB by running the "db/db.pl ESHOST:ESPORT upgrade" script. Make sure the number of ES nodes it sees is correct. You only need to do this on ONE ES node.
  • Once the cluster goes YELLOW and GREEN and the only unavailable shards are tags*, stats*, sequence run the following command. This will cause ES to distribute the current shards.
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable_allocation": false}}'
  • If there are many shards that need to be redistributed the defaults might take days, which is good for the cluster, but it might make you crazy. Increase the speed from default 3 streams at 20mb (60mb/sec) to something higher like 6 streams at 50mb (300mb/sec). Obviously adjust for speed of the new nodes' disks and network.
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{
  "indices.recovery.concurrent_streams":6, 
  "indices.recovery.max_bytes_per_sec":"50mb"}}'

Capture/Viewer nodes

Assuming you have identical/similar gear, you don't have to build everything from scratch.

  • Create a tarball of deps from your current build.
tar cvfz /tmp/moloch_and_nodejs.tgz --exclude /data/moloch/elasticsearch* --exclude '/data/moloch/data' /data/moloch/
  • Copy tarball to machines
scp /tmp/moloch_and_nodejs.tgz newmachine:/tmp
  • Extract the moloch_and_nodejs.tgz onto your new machine.
tar xvf /tmp/moloch_and_nodejs.tgz
  • Start new nodes
/data/moloch/bin/run_viewer.sh  (either from inittab, your service start script, or screen session loop)
/data/moloch/bin/run_capture.sh

Moloch Wiki

Add a custom sidebar

Clone this wiki locally