This project provides a client API for accessing the profiles generated by the Metron Profiler. This includes both a Java API and Stellar API for accessing the profile data. The primary use case is to extract profile data for use during model scoring.
The following are usage examples that show how the Stellar API can be used to read profiles generated by the Metron Profiler. This API would be used in conjunction with other Stellar functions like MAAS_MODEL_APPLY to perform model scoring on streaming data.
These examples assume a profile has been defined called 'snort-alerts' that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following.
{
"profiles": [
{
"profile": "snort-alerts",
"foreach": "ip_src_addr",
"onlyif": "source.type == 'snort'",
"update": { "s": "STATS_ADD(s, 1)" },
"result": "STATS_MEAN(s)"
}
]
}
During model scoring the entity being scored, in this case a particular IP address, will be known. The following examples highlight how this profile data might be retrieved.
Retrieve all values of 'snort-alerts' from '10.0.0.1' over the past 4 hours.
PROFILE_GET('snort-alerts', '10.0.0.1', 4, 'HOURS')
Retrieve all values of 'snort-alerts' from '10.0.0.1' over the past 2 days.
PROFILE_GET('snort-alerts', '10.0.0.1', 2, 'DAYS')
If the profile had been defined to group the data by weekday versus weekend, then the following example would apply.
Retrieve all values of 'snort-alerts' from '10.0.0.1' that occurred on 'weekdays' over the past month.
PROFILE_GET('snort-alerts', '10.0.0.1', 1, 'MONTHS', 'weekdays')
WARNING
By default, the Profiler creates Profiles with a period duration of 15 minutes. This means that data is accumulated, summarized and flushed every 15 minutes. The Client API must also have knowledge of this duration to correctly retrieve the profile data. If the client API is expected 15 minute periods, it will not be able to read data generated by a Profiler that has been configured with a 1 hour period.
The period duration can be configured in the Profiler by altering the Profiler topology's static properties file. The Stellar Client API currently provides no means to configure the period duration and defaults also to 15 minutes. This means that the Stellar Client API can only read profiles with a period duration of 15 minutes. This is a known limitation that has not yet been addressed.
These instructions step through the process of using the Stellar Client API on a live cluster. These instructions assume that the 'Getting Started' instructions included with the Metron Profiler have been followed. This will create a Profile called 'test' whose data will be retrieved with the Stellar Client API.
To validate that everything is working, login to the server hosting Metron. We will use the Stellar Shell to replicate the execution environment of Stellar running in a Storm topology, like Metron's Parser or Enrichment topology. Replace 'node1:2181' with the URL to a Zookeeper Broker.
[root@node1 0.2.1BETA]# bin/stellar -z node1:2181
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
[Stellar]>>> ?PROFILE_GET
Functions loaded, you may refer to functions now...
PROFILE_GET
Description: Retrieves a series of values from a stored profile.
Arguments:
profile - The name of the profile.
entity - The name of the entity.
durationAgo - How long ago should values be retrieved from?
units - The units of 'durationAgo'.
groups - Optional - The groups used to sort the profile.
Returns: The profile measurements.
[Stellar]>>> PROFILE_GET('test','192.168.138.158', 1, 'HOURS')
[12078.0, 8921.0, 12131.0]
The client API call above has retrieved the past hour of the 'test' profile for the entity '192.168.138.158'.