Automatic observables extraction from analysis reports.

[thevidocq]

EDIT THIS TITLE BEFORE POSTING. Use this template for bug reports. If you'd like to request a feature, please be as descriptive as possible and delete the template except the first section (Request Type)

Request Type

Question :)

Work Environment

Ubuntu 16.04 / TheHive:
TheHive3.0.10
Elastic4Play1.4.6
Play2.6.7
Elastic4s5.6.0
ElasticSearch5.6.2

Ubuntu16.04/Cortex:
Cortex2.0.4
Elastic4Play1.5.0
Play2.6.12
Elastic4s5.6.0
ElasticSearch client5.6.2
ElasticSearch cluster5.6.9

Problem Description

On analyzer configuration (es: cuckoo /fileinfo/msgparser) I have set TRUE the parameter "Extract observables" (Set to True to enable automatic observables extraction from analysis reports.)

But when the analysis finish I cannot see nothing about automatic extraction on TheHive case. I don't know if this behavior is expected or is a misconfiguration. Can you help me? How this feature work? Thanks.

[obikao]

I'm experiencing this issue as well.

The analyzers are set to extract observables but do not show up in the Hive case.

A good example is the Robtex Reverse Passive DNS. I ran it on an IP. It shows 10 different domains associated with this IP but did not create additional observables.

To add on to this, the data is being extracted in Cortex, in the raw report; however it looks like this data is not being passed on to TheHive

[saadkadhi]

When you enable observable extraction, the analyzers will try to figure out elements that look like observables in the results and add them to the resulting report. It uses ioc-parser and is prone to a lot of FPs (we have ideas to drop the number of FPs in the future).

As such, the extracted observables will be in the raw report that you can view in TheHive or in Cortex. However, this does not mean that TheHive will automatically add those potential observables to the case.

You are probably looking for TheHive-Project/TheHive#209 which should be included in TheHive 3.1 (end of July).

[3c7]

Remark: not ioc-parser but a selfmade regex-based extractor, because ioc-parser depends on some python 2 modules.

[obikao]

Additional Info:

I'd like to continue using Robtex_Reverse_PDNS_Query_1_0 plugin as an example. When viewing the data in the hive via the Raw Report function, it is only showing the "full" portion of the data structure (where the full data structure includes ['summary', 'full', 'success', 'artifacts'] as top level keys). I am assuming 'artifacts' is the key for the automatic observable extraction.

[3c7]

Yes, you're right. artifacts is the key used for automatic observable extraction. E.g. MISP uses it for receiving results from Cortex. Currently it is not possible to use the artifacts-part in TheHive itself. The adding of new observables through analyzer should be implemented in TheHive v3.1, as @saadkadhi said.

[obikao]

@thevidocq (or anyone that comes across this thread in the future) A hack for now is to do the following:

in TheHive/thehive-cortex/app/connectors/cortex/services/CortexSrv.scala change line 280 from
case _ ⇒
to
case _ ⇒ s'/path/to/a/python/script $cortexJobId $jobId'

also don't forget to add import sys.process._ to the imports

you can then write a quick script to do the work for you. I wrote an example here

https://github.com/obikao/random/blob/master/artifactParsing.py