Repository navigation
WISE
WISE is a framework for integrating data feeds into Moloch. The data feeds can be sourced from local files, remote URLs, or commercial services such as OpenDNS, Emerging Threats Pro, and others. The data feeds can set any Moloch field or even create new Moloch fields. Think of WISE as the next and better version of the tagger plugin.
WISE requires a plugin be installed on each moloch-capture instance, another plugin for each viewer instance and that a wiseService process be running. The wiseService process can be shared by multiple moloch-capture servers, even if they are in different moloch clusters. Choosing the machine that the wiseService runs on is important and the networking setup is crucial. The moloch-capture process connects to the wiseService AND the wiseService will need to reach out to any commercial services (if configured). All lookups are double cached, first in the wiseService so the remote service isn't queried too often and then in the moloch-capture process for load reduction. Maximum cache times and number of items are configurable.
WISE was first available with Moloch 0.11.3
The wiseService is the proxy and aggregator between moloch-capture and the various data sources. All the moloch-capture processes need to be able to reach it. If using external or commercial services then wiseService also needs to be able to reach those services. So pay attention to the networks available and machine setup.
- Pick a host on the correct networks and checkout moloch, wiseService lives in
moloch/capture/plugins/wiseService - Make sure node is installed
- run
npm install - Copy the sample ini file and edit it
cp wiseService.ini.sample wiseService.ini - Run wiseService
node wiseService.js -c wiseService.ini
Most settings will be in the individual sources for wise, however there are a few global settings
WISE uses multiple caches to speed up queries.
The wise.so plugin caches all results returned by wiseService, documented here. This cache will have all recent results, no matter the wise data source, so that the capture process doesn't need to communicate with wiseService for reoccurring traffic.
The wiseService also caches all results returned by external sources. All caching is done in memory, however since 0.15 Redis can be used for a larger external cache. An external source is something like OpenDNS or Reverse DNS, where it is impossible to load the entire data set into memory and WISE needs to make a external query to obtain results. An internal source is something like file or url, where each wiseService can have the entire data set.
Create a [cache] section
| Setting | Default | Description |
| type | memory |
memory or redis are supported
|
| url | empty |
For cache engines this is the url to connect to. For redis the format is [redis:]//[[user][:password@]]host:port[/db-number]
|
Each moloch-capture node needs to have the wise.so plugin
- Do a normal moloch update and compile
- In the config.ini file for moloch-capture add
- default section
wiseHost=wisehost.com - to the
pluginsline, either globally, per class or per node, addwise.so
- default section
Usually just setting wiseHost is enough. These can be set globally, per class or per node.
| Setting | Default | Description |
| wiseCacheSecs | 600 | Number of seconds to cache results before asking wiseService again |
| wiseHost | 127.0.0.1 | Host to connect to for wiseService |
| wiseMaxCache | 100000 | Max number of items to store in the wise cache that is local to each moloch-capture node |
| wiseMaxConns | 10 | Number of connections to wiseService, this is also the number of concurrent wise queries. |
| wiseMaxRequests | 100 | Number of oustanding requests to the wiseService |
| wisePort | 8081 | Port the wiseService is listening on |
Requires that wise.js be added to the viewerPlugins= line globally, per class or per node.
All sources support some common settings such excluding IPs, Domains and Email addresses from lookups. It is also possible to exclude across all sources by placing the exclude config in the wiseService section.
| Setting | Default | Description |
| excludeIPs | EMPTY | Semicolon separated list of IPs or CIDRs to exclude in lookups |
| excludeDomains | EMPTY | Semicolon separated list of modified glob patterns to exclude in lookups |
| excludeEmails | EMPTY | Semicolon separated list of modified glob patterns to exclude in lookups |
| cacheAgeMin | 60 | Number of minutes items in the cache for this source are valid for. Ignored for sources that use internal data, such as file sources. |
- To match zero or more characters either
*or%may be used - To match a single digit
#may be used - To match a character with special meaning, you must precede it with "~". The "~" also serves as the escape character
- To match one of a specific list of characters, you may surround the character with square brackets
- Globs are anchorced on both end, use
*to unanchor - String case doesn't matter
The Alien Vault data source currently uses the downloadable database that is updated often. Requires that access be purchased and configured.
Create a [alienvault] section to configure.
| Setting | Default | Description |
| key | REQUIRED | The API key |
The Emerging Threats Pro data source currently uses the downloadable database that is updated often. Requires that access be purchased and configured.
Create a [emergingthreats] section to configure.
| Setting | Default | Description |
| key | REQUIRED | The API key |
The OpenDNS source currently uses the bulk query api and does live queries. Requires that access be purchased and configured.
Create a [opendns] section to configure.
| Setting | Default | Description |
| key | REQUIRED | The API key |
| cacheSize | 200000 | Maximum number of results to cache |
The ThreatQ export interval time should be configured depending on process requirements and indicator volume. A minimum of version 1.16.4 is required for Moloch integration.
Create a [threatq] section to configure.
| Setting | Default | Description |
| key | REQUIRED | The API key |
| host | REQUIRED | Server hostname location |
The Threatstream data source currently uses the downloadable database that is only updated once a day. Requires that access be purchased and configured.
Create a [threatstream] section to configure.
| Setting | Default | Description |
| key | REQUIRED | The API key |
| user | REQUIRED | The API user |
The wiseService can monitor multiple files. Each file needs to have its own section, with the section name starting with file:. The wiseService automatically notices if the file changes and reloads it.
Create a [file:UNIQUENAME] section to configure.
| Setting | Default | Description |
| file | REQUIRED | The file to load |
| tags | REQUIRED | Comma separated list of tags to set for matches |
| type | REQUIRED | ip,domain,md5,email - The type of data in the file |
| format | csv | csv,tagger,json - The format of data file |
| column | 0 | For csv formatted files, which column is the data |
Since 0.15.1 Redis can be used as a wise data source.
If using redis as a wise cache you'll probably want to use a different redis "database" by specifying the database number in the url. Example: url=redis://localhost/1
Each redis source can only handle one type of data, although multiple redis sources can be configured and they can use the same redis database.
Create a [redis:UNIQUENAME] section to configure.
| Setting | Default | Description |
| url | REQUIRED |
The format is [redis:]//[[user][:password@]]host:port[/db-number]
|
| tags | REQUIRED | Comma separated list of tags to set for matches |
| type | REQUIRED | ip,domain,md5,email - The type of data in the file |
| format | csv | csv,tagger,json - The format of data file |
| column | 0 | For csv formatted files, which column is the data |
| template | %key% | The template when forming the key name. %key% = the key being looked up, %type% = the type being looked up. |
For IPs that are included by the ips setting, do a reverse lookup and place everything before the first dot in the field specified.
Create a [reversedns] section to configure.
| Setting | Default | Description |
| field | REQUIRED | The field to set with the hostname |
| ips | REQUIRED | Semicolon separated list of IPs or CIDRs to lookups |
| stripDomains | EMPTY |
If EMPTY then all domains are stripped after the FIRST period. When set ONLY domains that match the semicolon separated list of domain names are modified, and only the matching part is removed. Those that don't match will be saved in full. The list is checked in order. A leading dot is recommended. For example stripDomains=.foo.example.com;.example.com will convert test1.foo.example.com to test1, test2.bar.example.com to test2.bar and finally test3.bar.com to test3.bar.com - Added in 0.11.4
|
| cacheSize | 200000 | Maximum number of results to cache |
The wiseService can monitor and download URL. Each url needs to have its own section, with the section name starting with url:. The wiseService can automatically download and reload the files.
Create a [url:UNIQUENAME] section to configure.
| Setting | Default | Description |
| url | REQUIRED | The URL to load |
| tags | REQUIRED | Comma separated list of tags to set for matches |
| type | REQUIRED | ip,domain,md5,email - The type of data in the file |
| format | csv | csv,tagger,json - The format of data file |
| column | 0 | For csv formatted files, which column is the data |
| refresh | -1 | How often in minutes to refresh the file, or -1 to never refresh it |
| headers | Semicolon separated list of headers to send in the URL request |
This are WISE sources that aren't really data sources.
Experimental "History of Observed Data Indictors" plugin. This watches all queries to WISE and sends a feed to a configured elasticsearch cluster with firstSeen, lastSeen, and VERY rough count metric. The elasticsearch cluster must version version 1.5 or above and requires the hobi.groovy script be installed in the elasticsearch-x.x.x/config/scripts directory on all nodes. (This directory might need to be created.)
Added in 0.11.4
Create a [hodi] section to configure.
| Setting | Default | Description |
| esHost | REQUIRED | The elasticsearch connection string, usually host:port |
This plugin monitors configured files for right-click actions to send to all the viewer instances. Each file needs to have its own section, with the section name starting with right-click:. The format of the monitored files is the same as https://github.com/aol/moloch/wiki/Settings#rightclick
Create a [right-click:UNIQUENAME] section to configure.
| Setting | Default | Description |
| file | REQUIRED | The file to load |
WISE is http query-able so you can verify what it knows.
Type
- ip
- md5
- domain
- url
/[sectionname]/[type]/[value]
Query a particular source
/[type]/[value]
Query all sources
/dump/[sectionname]
Display all elements for a particular source
- When configured, if wiseService is down, moloch-capture will not start (current running are fine)
- When using moloch-capture in offline mode UDP/ICMP traffic isn't always tagged fully (Issue #311)