Forward indicators to STIX-Shifter

[0snap]

📔 Description

This PR adds a new app that connects with Threat Bus via ZeroMQ and subscribes to the topic stix2/indicator. When receiving indicators, the app calls out to the STIX-Shifter library to transform the STIX-2 Indicator to a specific query, depending on the user-provided configuration file. Users can configure the modules section in the config.yaml file. That section instructs the app to call out to certain STIX-Shifter modules, e.g., Splunk or Qradar.

This PR also extracts a template to create Python applications that connect to Threat Bus via ZeroMQ.

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/threatbus, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

Commit-by-commit

I tested this interactively with Splunk. The example config works for local Splunk installations. Grab the trial version from the official pages and start it. Then start Threat Bus with the zmq-app plugin enabled as usual and finally start this app. To see results, you first need to ingest data into Splunk (e.g., by uploading a file). Once that is done, you can use the tests/utils/zmq_sender.py to send specially crafted indicators (patterns) to match you data in Splunk. A pattern that worked well for me is pattern="[ipv4-addr:value = '192.168.1.103'] START t'2008-10-08T23:59:00.000Z' STOP t'2021-10-08T23:59:00.000Z'".

Unit / integration tests will be part of another PR. Same goes for the actual processing of result data.

[mavam]

Overall, this looks pretty nice. More details below.

STIX shifter error message upon CTRL+C

When hitting CTRL+C in the bridge, I get this error:

Unhandled error in exception handler
context: {'message': 'Task exception was never retrieved', 'exception': KeyboardInterrupt(), 'future': <Task finished name='Task-1' coro=<start() done, defined at /Users/mavam/code/tenzir/threatbus/apps/stix-shifter/stix_shifter_threatbus/shifter.py:186> exception=KeyboardInterrupt()>}
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 1744, in call_exception_handler
    self._exception_handler(self, context)
  File "/Users/mavam/code/tenzir/threatbus/apps/stix-shifter/stix_shifter_threatbus/shifter.py", line 221, in exception_handler
    logger.error(f"Error in async task: {context}")
  File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1475, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 954, in handle
    self.emit(record)
  File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1186, in emit
    self.stream = self._open()
  File "/usr/local/Cellar/python@3.8/3.8.9/Frameworks/Python.framework/Versions/3.8/lib/python3.8/logging/__init__.py", line 1176, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
NameError: name 'open' is not defined

Is this something we can try-catch?

splunk query

I'm getting this output in Threat Bus:

2021-05-18 09:50:26 DEBUG    [stix-shifter-threatbus] Calling Threat Bus management endpoint localhost:13370
2021-05-18 09:50:26 INFO     [stix-shifter-threatbus] Subscribing to topic 'stix2/indicator'...
2021-05-18 09:50:26 INFO     [stix-shifter-threatbus] Subscription successful. New p2p_topic: nveqctylybxlqxtijrorbrmwnkwmievl
2021-05-18 09:50:26 INFO     [stix-shifter-threatbus] Receiving via ZMQ on topic 127.0.0.1:13371/nveqctylybxlqxtijrorbrmwnkwmievl
2021-05-18 09:50:27 DEBUG    [stix-shifter-threatbus] Converting indicator from Threat Bus to module-specific query: {
    "type": "indicator",
    "spec_version": "2.1",
    "id": "indicator--783f7d87-afd0-4653-8b97-0f7567c127c6",
    "created": "2021-05-18T09:48:57.000Z",
    "modified": "2021-05-18T07:50:26.936642Z",
    "pattern": "[ipv4-addr:value = '198.189.255.75']",
    "pattern_type": "stix",
    "pattern_version": "2.1",
    "valid_from": "2021-05-18T07:50:26.936642Z"
}
2021-05-18 09:50:27 DEBUG    [stix-shifter-threatbus] Translated pattern to splunk query: {'queries': ['search ((src_ip = "198.189.255.75") OR (dest_ip = "198.189.255.75")) earliest="-5minutes" | head 10000 | fields src_ip, src_port, src_mac, src_ipv6, dest_ip, dest_port, dest_mac, dest_ipv6, file_hash, user, url, protocol']}

But not more. The first search works and delivers results. But the entire query doesn't. The problem is earliest="-5minutes". Once I manually add this to my splunk search, I get no more results.

App Template

I would be nice to make the fields that should be customized stand out more clearly, e.g., version etc.

[0snap]

@mavam I added logic for gracefully shutting down all ZeroMQ apps, that should elimitnate your stacktrace when hitting ctrl+c.
As for the earliest=-5minutes: that comes from STIX-Shifter. You can circumvent that by manually adding a validity time window to you Indicators you send for testing the app, e.g., a STIX-pattern like this:

"[ipv4-addr:value = '192.168.1.103'] START t'2008-10-08T23:59:00.000Z' STOP t'2021-10-08T23:59:00.000Z'"

[mavam]

Works as expected now. Really nice work!