WISE

Table of Contents WISE - With Intelligence See Everything Installation wiseService wiseService Settings Caching wise.so plugin wise.so Settings viewer Common Source Settings Glob Rules Commercial Sources Alien Vault Emerging Threats Pro OpenDNS Umbrella ThreatQ Threatstream Splunk Free Sources Elasticsearch File CSV Example Tagger Example JSON Example Subnets Example Redis ReverseDNS URL Other Sources HODI right-click What does WISE know? Query Source Query all Sources Display all values for Source Known Issues

WISE - With Intelligence See Everything

WISE is a framework for integrating data feeds into Moloch. The data feeds can be sourced from local files, remote URLs, or commercial services such as OpenDNS, Emerging Threats Pro, and others. The data feeds can set almost any Moloch field or even create new Moloch fields. Think of WISE as the next and better version of the tagger plugin.

WISE requires a plugin be installed on each moloch-capture instance, another plugin for each viewer instance and that a wiseService process be running. The wiseService process can be shared by multiple moloch-capture servers, even if they are in different moloch clusters. Choosing the machine that the wiseService runs on is important and the networking setup is crucial. The moloch-capture process connects to the wiseService AND the wiseService will need to reach out to any commercial services (if configured). All lookups are double cached, first in the wiseService so the remote service isn't queried too often and then in the moloch-capture process for load reduction. Maximum cache times and number of items are configurable.

WISE was first available with Moloch 0.11.3

Installation

wiseService

The wiseService is the proxy and aggregator between moloch-capture and the various data sources. All the moloch-capture processes need to be able to reach it. If using external or commercial services then wiseService also needs to be able to reach those services. So pay attention to the networks available and machine setup.

Pick a host on the correct networks and checkout moloch, wiseService lives in /data/moloch/wiseService
Initial install can be done with /data/moloch/bin/Configure --wise starting with Moloch 0.20.2

wiseService Settings

Most settings will be in the individual sources for wise, however there are a few global settings

Caching

WISE uses multiple caches to speed up queries.

The wise.so plugin caches all results returned by wiseService, documented here. This cache will have all recent results, no matter the wise data source, so that the capture process doesn't need to communicate with wiseService for reoccurring traffic.

The wiseService also caches all results returned by external sources. All caching is done in memory, however since 0.15 Redis can be used for a larger external cache. An external source is something like OpenDNS or Reverse DNS, where it is impossible to load the entire data set into memory and WISE needs to make a external query to obtain results. An internal source is something like file or url, where each wiseService can have the entire data set.

Create a [cache] section

Setting	Default	Description
type	memory	`memory` or `redis` are supported
url	empty	For cache engines this is the url to connect to. For redis the format is `[redis:]//[[user][:password@]]host:port[/db-number]`

wise.so plugin

Each moloch-capture node needs to have the wise.so plugin

Do a normal moloch update and compile
In the config.ini file for moloch-capture add
- default section wiseHost=wisehost.com
- to the plugins line, either globally, per class or per node, add wise.so

wise.so Settings

Usually just setting wiseHost is enough. These can be set globally, per class or per node.

Setting	Default	Description
wiseCacheSecs	600	Number of seconds to cache results before asking wiseService again
wiseHost	127.0.0.1	Host to connect to for wiseService, not used if wiseURL is set
wiseMaxCache	100000	Max number of items to store in the wise cache that is local to each moloch-capture node
wiseMaxConns	10	Number of connections to wiseService, this is also the number of concurrent wise queries.
wiseMaxRequests	100	Number of oustanding requests to the wiseService
wisePort	8081	Port the wiseService is listening on, not used if wiseURL is set
wiseTcpTupleLookups	FALSE	Should we send tcp tuple lookups to wise
wiseUdpTupleLookups	FALSE	Should we send udp tuple lookups to wise
wiseURL		(Since 1.5.0) The url to use to connect to wise

Since 1.5.0 wise also now lets you configure which fields are used for the standard wise types and you can add your own wise types. You do this by creating a [wise-types] section in the capture config.ini AND listing the fields using {type}={expression};{db:dbfield}.... The type field must be less then 12 characters, and is the same type field you would use in the wise service.

Standard config

Type	Default	Notes
ip	db:http.xffIp	srcIp and dstIp are always looked up for ip
url	db:http.uri
domain	db:http.host;db:dns.host
md5	db:http.md5;db:email.md5
sha256	db:http.sha256;db:email.sha256	supportSha256 must be set to true in your config file
email	db:email.src;db:email.dst
ja3	db:tls.ja3

viewer

Requires that wise.js be added to the viewerPlugins= line globally, per class or per node.

Common Source Settings

All sources support some common settings such excluding IPs, Domains and Email addresses from lookups. It is also possible to exclude across all sources by placing the exclude config in the wiseService section.

Setting	Default	Description
excludeIPs	EMPTY	Semicolon separated list of IPs or CIDRs to exclude in lookups
excludeDomains	EMPTY	Semicolon separated list of modified glob patterns to exclude in lookups
excludeEmails	EMPTY	Semicolon separated list of modified glob patterns to exclude in lookups
cacheAgeMin	60	Number of minutes items in the cache for this source are valid for. Ignored for sources that use internal data, such as file sources.
onlyIPs	EMPTY	If set, only ips that match the semicolon separated list of IPs or CIDRs will be looked up
fields	EMPTY	A "\n" separated list of fields that this source will add. Some wise sources automatically set for you. See TaggerFormat for more information on the parts of a field entry.
view	EMPTY

The view to show in session detail when openning up a session with unique fields. The value for view can either be writen in simplied format or in more powerful jade format. For the jade format TaggerFormat for more information excepteverything has to be on one line, so replace newlines with \n. Simple format looks like * To match zero or more characters either <code>* or % may be used

To match a single digit # may be used
To match a character with special meaning, you must precede it with "~". The "~" also serves as the escape character
To match one of a specific list of characters, you may surround the character with square brackets
Globs are anchorced on both end, use * to unanchor
String case doesn't matter

Commercial Sources

Alien Vault

The Alien Vault data source currently uses the downloadable database that is updated often. Requires that access be purchased and configured.

Create a [alienvault] section to configure.

Setting	Default	Description
key	REQUIRED	The API key

Emerging Threats Pro

The Emerging Threats Pro data source currently uses the downloadable database that is updated often. Requires that access be purchased and configured.

Create a [emergingthreats] section to configure.

Setting	Default	Description
key	REQUIRED	The API key

OpenDNS Umbrella

The OpenDNS source currently uses the bulk query api and does live queries. Requires that access be purchased and configured.

Create a [opendns] section to configure.

Setting	Default	Description
key	REQUIRED	The API key
cacheSize	200000	Maximum number of results to cache

ThreatQ

The ThreatQ export interval time should be configured depending on process requirements and indicator volume. A minimum of version 1.16.4 is required for Moloch integration.

Create a [threatq] section to configure.

Setting	Default	Description
key	REQUIRED	The API key
host	REQUIRED	Server hostname location

Threatstream

The Threatstream data source currently uses the downloadable database that is only updated once a day. Requires that access be purchased and configured.

Create a [threatstream] section to configure.

Setting	Default	Description
key	REQUIRED	The API key
user	REQUIRED	The API user

Splunk

The splunk wise service can run in two different modes. It can query splunk for every value or it can query splunk periodicly for a table of values. Many splunk operators prefer the periodic query since they can scale for it.

Create a [splunk:UNIQUENAME] section to configure.

Setting	Default	Description
type	REQUIRED	any type- The type of data in the file
host	REQUIRED	The splunk hostname
keyColumn	REQUIRED	The column to use from the returned data to use as the key
periodic	false	Should we do periodic queries or individual queries
port	REQUIRED	The splunk port
query	REQUIRED	The query to run against splunk. For non periodic queries the string %%SEARCHTERM%% will be replaced with the key.
version	5	The splunk api version to use

Example config that will query splunk for all the vpn_ip to user name mappings during the last 24 hours every 60 seconds. It will then set the user field for any ip that matches.

&#91;splunk&#58;users&#93;
type &#61; ip
format &#61; json
host &#61; spunk.example.com
port&#61;5500
username&#61;theuser
password&#61;thepassword
periodic&#61;60
query&#61;search index&#61;&quot;THEINDEX&quot; sourcetype&#61;&quot;vpn&quot; assigned earliest&#61;&#45;24h &#124; rex &quot;User &lt;(?&lt;user&gt;&#91;^&gt;&#93;+)&gt;.&#42;IPv4 Address &lt;(?&lt;vpn_ip&gt;&#91;^&gt;&#93;+)&gt;&quot; &#124; dedup vpn_ip &#124; table user, vpn_ip
keyColumn&#61;vpn_ip
fields&#61;field&#58;user&#59;shortcut&#58;user

Free Sources

Elasticsearch

The elasticsearch wise service can query elasticsearch for fields to set

Create a [elasticsearch:UNIQUENAME] section to configure.

Setting	Default	Description
type	REQUIRED	The type of data in the file, such as ip,domain,md5,ja3,email, or something defined in `[wise-types]`
elasticsearch	REQUIRED	Elasticsearch base url
esIndex	REQUIRED	The index pattern to look at
esTimestampField	REQUIRED	The field to use in queries that has the timestamp in ms.
esMaxTimeMS	1 hour	Timestamp field must be less then this
esResultField	REQUIRED	Field that is required to be in the result

Example config that will query elasticsearch for an ip that is in the 10.172/16 space, in the index TheIndex-*, only looking at records that have a @timestamp field newer tehn 86400000ms. It looks at the cef_ext.src field and only looks at records that has a cef_ext.suser field set. Once it has a result it sets the user field in moloch to whatever the cef_ext.suser field is in the document.

type&#61;ip
onlyIPs&#61;10.172.0.0/16
elasticsearch&#61;http&#58;//ELKCLUSTERHOST1&#58;9200,http&#58;//ELKCLUSTERHOST2&#58;9200
esIndex&#61;TheIndex&#45;&#42;
esTimestampField&#61;@timestamp
esQueryField&#61;cef_ext.src
esMaxTimeMS&#61;86400000
esResultField&#61;cef_ext.suser
fields&#61;field&#58;user&#59;shortcut&#58;cef_ext.suser

File

The wiseService can monitor multiple files. Each file needs to have its own section, with the section name starting with file:. The wiseService automatically notices if the file changes and reloads it.

Create a [file:UNIQUENAME] section to configure.

Setting	Default	Description
file	REQUIRED	The file to load
tags	REQUIRED	Comma separated list of tags to set for matches
type	REQUIRED	The type of data in the file, such as ip,domain,md5,ja3,email, or something defined in `[wise-types]`
format	csv	csv,tagger,json - The format of data file
keyColumn	0	For json formatted files, which json field is the key
column	0	For csv formatted files, which column is the data

CSV Example

Config File

&#91;file&#58;ipcsv&#93;
file&#61;./ip.wise.csv
tags&#61;ipwisecsv
type&#61;ip
column&#61;1
format&#61;csv
&#35;Asset field already exist, use field 0 for value.  extra field is new, use field 2 for value
fields&#61;field&#58;asset&#59;shortcut&#58;0\nfield&#58;extra&#59;kind&#58;lotermfield&#59;count&#58;true&#59;friendly&#58;extra&#59;db&#58;extra&#59;help&#58;Help for Extra&#59;shortcut&#58;2

The CSV File

blah1,10.0.0.3
blah2,10.0.0.2,foo

Tagger Example

Config File

&#91;file&#58;iptagger&#93;
file&#61;./ip.wise.tagger
tags&#61;ipwisetagger
type&#61;ip
format&#61;tagger

The Tagger File

&#35;field&#58;extra&#59;kind&#58;lotermfield&#59;count&#58;true&#59;friendly&#58;extra&#59;db&#58;extra&#59;help&#58;Help for Extra
10.0.0.3&#59;asset&#61;blah1
10.0.0.2&#59;asset&#61;blah2&#59;extra&#61;foo

JSON Example

Config File

&#91;file&#58;ipcsv&#93;
file&#61;./ip.wise.json
tags&#61;ipwisejson
type&#61;ip
keyColumn&#61;theip
format&#61;json
&#35;Asset field already exist, use field asset for value.  extra field is new, use field extra for value
fields&#61;field&#58;asset&#59;shortcut&#58;asset\nfield&#58;extra&#59;kind&#58;lotermfield&#59;count&#58;true&#59;friendly&#58;extra&#59;db&#58;extra&#59;help&#58;Help for Extra&#59;shortcut&#58;extra\n

The JSON File

&#91;&#123;&quot;asset&quot;&#58; &quot;blah&quot;, &quot;theip&quot;&#58; &quot;10.0.0.3&quot;&#125;,
 &#123;&quot;asset&quot;&#58; &quot;blah2&quot;, &quot;theip&quot;&#58; &quot;10.0.0.2&quot;, &quot;extra&quot;&#58; &quot;foo&quot;&#125;&#93;

note: You use shortcut to match between fields in the JSON dictionary and the properties in ElasticSearch.

Subnets Example

More complex example of the above where you want to create a new section. This example defines t

&#91;file&#58;subnets&#93;
type&#61;ip
format&#61;json
file&#61;subnets.json
keyColumn&#61;ip
fields&#61;field&#58;subnets.description&#59;kind&#58;termfield&#59;count&#58;true&#59;friendly&#58;Description&#59;db&#58;subnets.description&#59;help&#58;Description&#59;shortcut&#58;description\nfield&#58;subnets.securityzone&#59;kind&#58;termfield&#59;count&#58;true&#59;friendly&#58;Security Zone&#59;db&#58;subnets.securityzone&#59;help&#58;Security Zone&#59;shortcut&#58;securityZone\nfield&#58;subnets.vlan&#59;kind&#58;integer&#59;count&#58;true&#59;friendly&#58;Vlan&#59;db&#58;subnets.vlan&#59;help&#58;Vlan&#59;shortcut&#58;vlan\nfield&#58;subnets.site&#59;kind&#58;termfield&#59;count&#58;true&#59;friendly&#58;Site&#59;db&#58;subnets.site&#59;help&#58;Site&#59;shortcut&#58;site
&#35; Jade view method
view&#61;if (session.subnets)\n    +arrayList(session.subnets, &#39;description&#39;, &#39;Description&#39;, &#39;subnets.description&#39;)\n    +arrayList(session.subnets, &#39;label&#39;, &#39;Label&#39;, &#39;subnets.label&#39;)\n    +arrayList(session.subnets, &#39;securityzone&#39;, &#39;Security Zone&#39;, &#39;subnets.securityzone&#39;)\n    +arrayList(session.subnets, &#39;vlan&#39;, &#39;Vlan&#39;, &#39;subnets.vlan&#39;)\n    +arrayList(session.subnets, &#39;site&#39;, &#39;Site&#39;, &#39;subnets.site&#39;)
&#35; Simple view method
&#35;view&#61;require&#58;subnets&#59;title&#58;Subnets&#59;fields&#58;subnets.description,subnets.label,subnets&#58;securityzone,subnets.vlan,subnets.site

The JSON File

&#91;&#123;&quot;description&quot;&#58; &quot;the description&quot;, &quot;label&quot;&#58; &quot;interesting label&quot;, &quot;securityzone&quot;&#58; &quot;hot&quot;, &quot;vlan&quot;&#58; 123, &quot;site&quot;&#58; &quot;secret&quot;,  &quot;ip&quot;&#58; &quot;10.0.0.3&quot;&#125;,
 &#123;&quot;description&quot;&#58; &quot;the description2&quot;, &quot;label&quot;&#58; &quot;interesting label2&quot;, &quot;securityzone&quot;&#58; &quot;cold&quot;, &quot;vlan&quot;&#58; 123, &quot;site&quot;&#58; &quot;secret&quot;,  &quot;ip&quot;&#58; &quot;10.0.0.2&quot;&#125;&#93;

Redis

Since 0.15.1 Redis can be used as a wise data source. If using redis as a wise cache you'll probably want to use a different redis "database" by specifying the database number in the url. Example: url=redis://localhost/1 Each redis source can only handle one type of data, although multiple redis sources can be configured and they can use the same redis database.

Create a [redis:UNIQUENAME] section to configure.

Setting	Default	Description
url	REQUIRED	The format is `[redis:]//[[user][:password@]]host:port[/db-number]`
tags	REQUIRED	Comma separated list of tags to set for matches
type	REQUIRED	The type of data in the file, such as ip,domain,md5,ja3,email, or something defined in `[wise-types]`
format	csv	csv,tagger,json - The format of data file
column	0	For csv formatted files, which column is the data
template	%key%	The template when forming the key name. %key% = the key being looked up, %type% = the type being looked up.

ReverseDNS

For IPs that are included by the ips setting, do a reverse lookup and place everything before the first dot in the field specified.

Create a [reversedns] section to configure.

Setting	Default	Description
field	REQUIRED	The field to set with the hostname
ips	REQUIRED	Semicolon separated list of IPs or CIDRs to lookups. Ips that don't match this list will NOT be reverse lookuped.
servers	EMPTY	Since 1.6.1, if set the reversedns source will use the semicolon separated list of ip addresses to reverse lookuped.
stripDomains	EMPTY	If EMPTY then all domains are stripped after the FIRST period. When set ONLY domains that match the semicolon separated list of domain names are modified, and only the matching part is removed. Those that don't match will be saved in full. The list is checked in order. A leading dot is recommended. For example `stripDomains=.foo.example.com;.example.com` will convert `test1.foo.example.com` to `test1`, `test2.bar.example.com` to `test2.bar` and finally `test3.bar.com` to `test3.bar.com` - Added in 0.11.4
cacheSize	200000	Maximum number of results to cache

URL

The wiseService can monitor and download URL. Each url needs to have its own section, with the section name starting with url:. The wiseService can automatically download and reload the files.

Create a [url:UNIQUENAME] section to configure.

Setting	Default	Description
url	REQUIRED	The URL to load
tags	REQUIRED	Comma separated list of tags to set for matches
type	REQUIRED	The type of data in the file, such as ip,domain,md5,ja3,email, or something defined in `[wise-types]`
format	csv	csv,tagger,json - The format of data file
column	0	For csv formatted files, which column is the data
refresh	-1	How often in minutes to refresh the file, or -1 to never refresh it
headers		Semicolon separated list of headers to send in the URL request

Other Sources

This are WISE sources that aren't really data sources.

HODI

Experimental "History of Observed Data Indicators" plugin. This watches all queries to WISE and sends a feed to a configured elasticsearch cluster with firstSeen, lastSeen, and VERY rough count metric. The elasticsearch cluster must version version 1.5 or above and requires the hobi.groovy script be installed in the elasticsearch-x.x.x/config/scripts directory on all nodes. (This directory might need to be created.)

Added in 0.11.4

Create a [hodi] section to configure.

Setting	Default	Description
esHost	REQUIRED	The elasticsearch connection string, usually host:port

right-click

This source monitors configured files for right-click actions to send to all the viewer instances that connect to this WISE Server. Each file needs to have its own section, with the section name starting with right-click:. The format of the monitored files is the same as https://github.com/aol/moloch/wiki/Settings#rightclick WISE will auto reload the right click files if they change.

Create a [right-click:UNIQUENAME] section to configure.

Setting	Default	Description
file	REQUIRED	The file to load

So for example you might have

&#91;right&#45;click&#58;virustotal&#93;
file&#61;/data/moloch/etc/rightclick&#45;virustotal.ini

and then /data/moloch/etc/rightclick-virustotal.ini could contain

VTIP&#61;url&#58;https&#58;//www.virustotal.com/en/ip&#45;address/%TEXT%/information/&#59;name&#58;Virus Total IP&#59;category&#58;ip
VTHOST&#61;url&#58;https&#58;//www.virustotal.com/en/domain/%HOST%/information/&#59;name&#58;Virus Total Host&#59;category&#58;host
VTURL&#61;url&#58;https&#58;//www.virustotal.com/latest&#45;scan/%URL%&#59;name&#58;Virus Total URL&#59;category&#58;url

Starting with Moloch 1.5, a section named [right-click] with NO colon or unique name, allows the right clicks to live in the wise.ini file, but require a wiseService restart to reload.

What does WISE know?

WISE is http query-able so you can verify what it knows.

Type

ip
email
md5
domain
url

Query Source

/[sectionname]/[type]/[value]

Query a particular source

Query all Sources

/[type]/[value]

Query all sources

Display all values for Source

/dump/[sectionname]

Display all elements for a particular source

Known Issues

When configured, if wiseService is down, moloch-capture will not start (current running are fine)
When using moloch-capture in offline mode UDP/ICMP traffic isn't always tagged fully (Issue #311)

Arkime Wiki

Uh oh!

WISE

Table of Contents

WISE - With Intelligence See Everything

Installation

wiseService

wiseService Settings

Caching

wise.so plugin

wise.so Settings

viewer

Common Source Settings

Commercial Sources

Alien Vault

Emerging Threats Pro

OpenDNS Umbrella

ThreatQ

Threatstream

Splunk

Free Sources

Elasticsearch

File

CSV Example

Tagger Example

JSON Example

Subnets Example

Redis

ReverseDNS

URL

Other Sources

HODI

right-click

What does WISE know?

Query Source

Query all Sources

Display all values for Source

Known Issues

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally