METRON-659 Emulate Sensors in Development Environments #417
Conversation
|
Forgot to mention, a side effect of this change is that Quick Dev takes half the time to deploy. For me, it went from about 31 minutes to 17 minutes. |
| # how many messages to send in each 'batch'. the messages are drawn randomly | ||
| # from the entire set of canned data. defaults to 1000. | ||
| # | ||
| COUNT=${2:-100} |
There was a problem hiding this comment.
This is default 100 not 1000, no? I noticed the same for the other stubs.
There was a problem hiding this comment.
Good catch. I'll fix it.
There was a problem hiding this comment.
Done. Updated these values so they get set by Ansible to the defaults set in the properties. This matches behavior elsewhere. The default in the Ansible properties is 10 messages every 2 seconds.
|
@nickwallen Nice contribution! The docs are solid, and I like the configurability of the batch size and frequency. And the quicker deploy time is definitely a plus. Is that because we're no longer deploying the sensors by default? I'm on the fence about how we should handle that. I originally thought it best to deploy and not start the real sensors. But if we're saving 14 minutes in the Vagrant deployment, I could be convinced otherwise. And the more I think about it, you could just run the Ansible role for the sensors which is easy enough. I don't really have any other suggested changes besides the minor remark above about the default of 100 vs 1000. |
|
Yes, quicker deploy is because we no longer build and deploy the sensors. The sensors are built from source, which takes a good chunk of time (and patience.) |
|
@nickwallen where does the mock data come from? Did you create is? Is there any concern for licensing? |
|
I ran the sensors against the example pcap file that gets deployed with Metron. I captured and 'canned' the textual output that gets pushed to Kafka. I did it this way so that the data produced by the sensor stubs is similar to what you will see when running with the sensors/pcap replay. No licensing issues. |
A service has been created to simulate the behavior of a sensor by sending canned telemetry data to a Kafka topic. These "Sensor Stubs" consume fewer resources than the actual sensor that they replace.
Testing
I have tested this change by deploying and validating the following environments. For each environment I manually validated it and ensured that data flowed end-to-end and was visible within the Metron dashboard.
Q&A
(Q) How do the sensor stubs work?
The stubs are installed with a set of canned data for each sensor type. A subset of this canned data is randomly selected and sent to the Kafka topic in batches. The timestamp of each message is updated to match current system time.
(Q) How do I configure the message rate?
The number of telemetry messages sent in each batch, along with the time delay between batches is configurable. Before installation, these values can be configured by redefining
sensor_stubs_delayandsensor_stubs_count. The values can also be configured by altering the deployed system service script at/etc/init.d/sensor-stubs.(Q) How do I install the sensor stubs?
Using the default playbooks, this role can be installed by using the Ansible tag
sensor-stubs. This service is installed on the same hosts where the sensors would be; defined by thesensorshost group.The defaults for the "Quick Dev" and "Full Dev" environments have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves. The Amazon EC2 environment continues to install the original sensors by default.
(Q) How do I use the sensor stubs?
Start all sensor stubs. The output includes the PID for each running sensor stub.
Stop all sensor stubs.
Start only the Bro sensor stub.
(Q) How do I install the original sensors?
The default behavior can be changed by skipping the
sensor-stubsflag and including thesensorsflag. For example, to deploy "Quick Dev" with the original sensors run the following command.