Skip to content
TNogisto edited this page Feb 6, 2014 · 94 revisions

Table of Contents

General

How do I reset Moloch?

First shutdown any running viewer or capture processes so no new data is recorded, leave elasticsearch running.

To delete all the SPI data stored in elasticsearch use the db.pl script with either the init or wipe command. The only different between the two commands is that wipe leaves the added users so they don't need to be re-added.

    /data/moloch/db/db.pl ESHOST:ESPORT wipe

Now you need to delete the pcap files, which are just stored on the file system in raw format. This you need to do on all the capture machines

    /bin/rm -f /data/moloch/raw/*

Elasticsearch

How many elasticsearch nodes or machines do I need?

The answer of course is "it depends". Factors include how much memory each box has, how many days you want to store meta data for, how fast the disks are, what percentage of the traffic is HTTP, the average transfer rate of all the interfaces, if the sessions are long lived or short lived, how fast response times should be for operators, how many operators are querying at the same time, etc.

A rough guideline for NODES is 1/4 * Number_of_Gigabit_Interfaces * Number_of_Days_of_History. If you have large machines then you can run multiple nodes per MACHINE.

The good news is it is easy to add new nodes in the future, so feel free to start with less nodes. As a temporary fix to capacity problems you can reduce the number of days of meta data that are stored. Just use the elastic head interface to delete the oldest sessions-* index.

Data never gets deleted

Make sure you set up the db/daily.sh script to run once a day, during off peak. This script optimizes the previous days data and deletes the oldest data.

When do I add additional nodes? Why are queries slow?

If queries are too slow then the easiest fix is to add additional elasticsearch nodes. You should also use the bigdesk http://moloches-machine:9200/_plugin/bigdesk/ and watch the heap memory graph, if it is approaching the max it is time to add more nodes or reduce the number of days being stored.

Elasticsearch doesn't do well if Java hits a OutOfMemory condition. If you ever have one you should immediate delete the oldest sessions-* index, update the daily.sh script to delete more often, and restart the elasticsearch cluster. Then you should order more machines. :)

Adding additional nodes

See detailed instructions in the Multiple-Host-HOWTO wiki page

How to enable elastic search replication

Turning on replication will consume twice the disk space on the nodes and increase the network bandwidth between nodes, so make sure you actually need replication.

To fix future days

  • Edit the db/db.pl file and change the number_of_replicas from 0 to 1
  • Run the "db/db.pl upgrade" script

Elasticsearch crashed and now won't return green

Make sure all the nodes are actually running by checking bigdesk. Before starting to scream, make sure to wait at least 5-10 minutes after restarting before giving up. The most likely cause is the _state directory is missing for various shards. If you can live without the ENTIRE day that isn't going green the easiest solution is to use elastic head to just delete the day, the sessions-* index. http://moloches-machine:9200/_plugin/head/

If you want to try and recover the rest of the day then you need

  • Find the machines with the missing index
  • Check to see if the shard even exists (For example shard 4 on 120715 would live in /disk1/data/Moloch/nodes/0/indices/sessions-120715/4)
  • If that directory is missing you need to create it by copying a empty version of the same directory from a temporary elastic search instance.
Upgrade to 0.20.6 or later which seems to fix this issue

How do I upgrade Elastic Search?

If using a multi node elastic search cluster you must shutdown the entire cluster and restart the entire cluster with the new version. Current versions of elastic search do not handle mismatch versions.

    cd elasticsearch-NEWVERSION
    bin/plugin -install mobz/elasticsearch-head
    bin/plugin -install lukas-vlcek/bigdesk
  • edit the ES start script so it has the correct version, /data/moloch/bin/run_es.sh (if using single hostconfig)
  • Shutdown current elasticsearch node
    curl -XPOST http://localhost:9200/_shutdown
  • Start the new version back up
    /data/moloch/bin/run_es.sh

Capture

Why am I dropping packets?

The most common reason is the disk IO subsystem can't keep up which is causing moloch-capture to pause and fall behind reading the packets off the network.

  • Don't run capture and elasticsearch on the same machine
  • Make sure you actually have enough disk write thru capacity and disks. For example for a 1G link with RAID 5 you need
    • At least 5 spindles if using a RAID 5 card with write cache enabled
    • At least 15 spindles (or more) if using a RAID 5 card with write cache disabled
  • Make sure your RAID card can actually handle the write rate, many onboard RAID 5 controllers can not handle sustained 1G write rates
  • Switch to RAID 0 from RAID 5 if you can live with the TOTAL data loss on a single disk failure
If using EMC for disks
  • Make sure write cache is enabled for the LUNs
  • If it is a CX with SATA drives, RAID-3 is optimized for large sequential I/O
  • Monitor EMC lun queue depth, may be too many hosts sharing it
To check your disk IO run iostat -xm 5 and look at the following
  • wMB/s will give you the current write rate, does it match up with what you expect?
  • avgqu-sz should be near or less then 1, otherwise linux is queueing instead of doing
  • await should be near or less then 10, otherwise the IO system is slow, which will slow moloch-capture down
Other things to do/check
  • If using RAID 5 make sure you have write cache enabled on the RAID card
    • Adaptec Example: arcconf SETCACHE 1 LOGICALDRIVE 1 WBB
  • Use taskset to give moloch-capture its own CPU
  • Check that the CPU you are giving moloch-capture isn't handling lots of interrupts (cat /proc/interrupts)
  • Make sure other processes aren't using the same CPU as moloch-capture
  • For ifdrop's increase the ring buffer using ethtool -g to view maximum and ethtool -G to set

How do I import existing PCAPs?

Think of the 'moloch-capture' binary much like you would 'tcpdump'. 'moloch-capture' can listen to live network interface(s), or read from historic packet capture files.

${moloch_dir}/bin/moloch-capture -c [config_file] -r [pcap_file]

For an entire directory, use -R [pcap]

See ${moloch_dir}/bin/moloch-capture --help for more info

How do I monitor multiple interfaces?

A capture process will only monitor a single interface. To monitor multiple interfaces on a single machine, you will need multiple capture processes.

  • Since by default moloch uses the unqualified hostname as the name of the moloch node, you'll need to come up with a naming scheme. Appending a, b, c, ... or the interface number to the hostname are possible methods.
  • edit the /data/moloch/etc/config.ini and create a section for each of the moloch nodes. Assuming the defaults are correct in the [default] section the only thing thing that MUST be set is the interface item. It is also common if running a cluster of elasticsearch nodes to have each moloch node talk to a different elasticsearch node.
 [moloch-m01a]
 interface=eth2
 [moloch-m01b]
 interface=eth5
  • If hostname on the machine doesn't return a FQDN you'll also need to set a viewUrl
  • create two start up scripts, you will now need to specify the moloch node name with the -n option and change the log files so they are separate.
 TDIR=/data/moloch
 cd ${TDIR}/bin
 /bin/rm -f ${TDIR}/capturea.log.old
 /bin/mv ${TDIR}/logs/capturea.log ${TDIR}/logs/capturea.log.old
 ${TDIR}/bin/moloch-capture -n moloch-m01a -c ${TDIR}/etc/config.ini > ${TDIR}/logs/capturea.log 2>&1

and

 TDIR=/data/moloch
 cd ${TDIR}/bin
 /bin/rm -f ${TDIR}/captureb.log.old
 /bin/mv ${TDIR}/logs/captureb.log ${TDIR}/logs/captureb.log.old
 ${TDIR}/bin/moloch-capture -n moloch-m01b -c ${TDIR}/etc/config.ini > ${TDIR}/logs/captureb.log 2>&1
  • Add both scripts to inittab or upstart

Moloch capture crashes

Please file a ticket on github with the stack trace.

  • You'll need to allow suid or user changing programs to save core dumps. Use sysctl to change until the next reboot, setting it to 0 will change it back to the default.
sysctl -w fs.suid_dumpable=2
  • Run moloch-capture and get it to crash
  • Look for the most recent core file
  • Run gdb (You may need to install the gdb package first)
gdb /data/moloch/bin/moloch-capture corefilename
  • Get the back trace using the "bt" command

Capture log is full of libnids errors at start

When starting Capture, capture log is full of errors like

Feb  * 1:24:45 nids.c:917 moloch_nids_syslog(): NIDSIP: Invalid IP header - Length issue, was packet truncated?

Disable rx-checksumming for the interface with ethtool.

ethtool -K eth0 rx off

After that the errors should be gone.

Viewer

Where do I learn more about the expressions available

Click on the owl

Exported pcap files are corrupt, sometimes session detail fails

  • Most common problem is that the timestamps between the moloch machines are different. Make sure ntp is running everywhere, or that the time stamps are in sync.

SPI View tab missing

The SPI View tab requires that elasticsearch version 0.90 RC2 or later be installed.

Map counts are wrong

  • The source and destination ips are each counted, so the map should total twice the number of sessions
  • Currently elasticsearch only has accurate counts up to 2 billion uniques
  • Some countries aren't shown, but can still be searched using their ISO-3

Browser security exception/replacing cert

By default the easybutton-singlehost.sh script creates a self signed cert. This will cause good browsers to throw a security exception.

Quick Fix

Most browsers have a method for adding a self signed cert so its accepted and stops showing warnings. Start with this google query https://www.google.com/search?q=adding+a+self+signed+certificate for suggestions on your browser

Correct Fix

  • Contact your pki folks and ask for a new cert, or purchase one
    • They may require a csr, you can use the /data/moloch/etc/moloch.csr file or create a new one
    • if you will have multiple viewer machines (or think you will) it is easier to give them ALL the host names at once so the single cert can have multiple alt names in it
  • Replace the /data/moloch/etc/moloch.crt and moloch.key files on all the viewer nodes
  • Restart the viewer (node) process

What browsers are supported?

Recent versions of Chrome, Firefox, and Safari should all work fairly equally. Development is done mostly with Chrome on a Mac so it gets the most attention. Firefox on Windows is used heavily by the analysts. Currently no testing or development is done with IE or Opera.

Viewer doesn't run after upgrading node.js

The packages that viewer depends on must be reinstalled after upgrading node.js on a machine.

cd /data/moloch/viewer
mv node_modules node_modules.save
npm cache clean
npm install

Error: getaddrinfo EADDRINFO

This seems to be caused when proxying requests from one viewer node to another and the machines don't use FQDNs for their hostnames and the short hostnames are not resolvable by DNS. You can check if your machine uses FQDNs by running the hostname command. There are several options to resolve the error

  1. Configure the OS to use FQDNs
  2. Make it so DNS can resolve the shortnames or add the shortnames to the hosts file
  3. Edit config.ini and add a viewUrl for each node. This part of the config file must be the same on all machines. (We recommend you just use the same config file everywhere). Example:
[node1_eth0]
interface=eth0
viewUrl=http://node1.fqdn
[node1_eth1]
interface=eth1
viewUrl=http://node1.fqdn
[node2]
interface=eth1
viewUrl=http://node2.fqdn

How do I proxy Moloch using Apache

Apache, and other web servers, can be used to provide authentication or other services for Moloch. While operators will use the proxy, the viewer processes still need direct access to each other.

  • Install apache, turn on the auth method of your choice, and make sure mod_ssl is loaded
  • Set up a SSL proxy
SSLProxyEngine On
ProxyRequests On
ProxyPass        /moloch/ https://localhost:8005/ retry=0
ProxyPassReverse /moloch/ https://localhost:8005/
  • Restart apache
  • Using the Moloch UI (by going directly to a non proxy moloch) make sure the "Web Auth Header" is checked for the users.
  • Edit Moloch's config.ini and tell it to use the REMOTE_USER header, or whatever header your auth package uses. For the config file use the all lower case version of the header.
userNameHeader=remote_user
  • Edit Moloch's config.ini and add a new section for the proxy host. Set the webBasePath to the path used above. All other sections should NOT have a webBasePath.
[moloch-proxy]
viewPort = 8005
webBasePath = /moloch/
  • Start the moloch-proxy viewer
  • To prevent the users from going directly to moloch in the future, scramble their passwords. You might want to leave an admin user that doesn't use the apache auth, or you can always temporarily add one with the addUser.js script.

Clone this wiki locally