Short Report is not shown on observables (3.0.8)

[crackytsi]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian
OS version (client)	Seven using Chrome Browser
TheHive version / git hash	3.0.6
Package Type	DEB

Problem Description

Unfortunately the short-report is still not always shown on observables (observables-tab and tab of the observable itself).

[nadouani]

Can you provide the JSON definition of the observable? Just to see if it's a data issue or a display issue

[crackytsi]

How can I retrieve this definiton?

[nadouani]

curl -XGET -H 'Authorization: Bearer API_KEY' 'http://SERVER:PORT/api/case/artifact/OBSERVABLE_ID'

[nadouani]

The OBSERVABLE_ID is the last part of the URL path of observable details page

[crackytsi]

{
   "_type" : "case_artifact",
   "updatedBy" : "user1",
   "createdAt" : 1520514455787,
   "createdBy" : "user1",
   "status" : "Ok",
   "data" : "192.168.200.2",
   "updatedAt" : 1520514472316,
   "_id" : "ea19684074eacd32ac5b1a087977fc30",
   "sighted" : false,
   "message" : "environment1",
   "reports" : {
      "HASDB_1_0" : {
         "taxonomies" : [
            {
               "predicate" : "Net",
               "level" : "info",
               "value" : "\"192.168.200.0/28 NPS FCARAS123\"",
               "namespace" : "HASDB"
            }
         ]
      }
   },
   "ioc" : false,
   "tlp" : 2,
   "dataType" : "ip",
   "id" : "ea19684074eacd32ac5b1a087977fc30",
   "tags" : [
      "environment1"
   ],
   "startDate" : 1520514455790
}

[crackytsi]

Hi,
As you might have noticed, this are analyzers that we use internal.
Strangely on the overview table, I can see Reports for 2 analyzers (HASDB and AssetDB), both Long reports are shown correctly.

[nadouani]

I'm taking a look :)

[nadouani]

Well all looks good, the data (reports attribute) is on the observable, well formed too.

Do you have any javascript error on your browsers console?

[crackytsi]

There are 2 reports.... not just 1.

The problem seems to be that not always all reports are added to the data structure.
No, I have no javascript error.

[nadouani]

Wait, wait. Your observables should have 2 reports but it's definition contains only one instead of 2?

In the observable details page, you see both mini reports, and in observables list you see only one mini report?

Just to be sure we are talking about the same thing

[crackytsi]

Hi,
Right, my observable should have 2 reports, but the Definition retrieved by the API Call listes only one.

In the observable page I see 2 reports, but there is also only one mini-reports.
If I click on the analyzer, I see both Long reports.

[nadouani]

and both analyzer jobs are successful right?

[nadouani]

This sounds like #409

[crackytsi]

Both Jobs were successfull.
yes it does, but this issue was solved and I'm using version 3.0.6 :-/

[nadouani]

Yes, I know, but it could has been solved partially, or you could have a corner case that has not been taken into account. The problem with this issue is that it's not easy to reproduce.

@To-om will take a look on it

[crackytsi]

If I can support you, let me know
The Case was created after upgrade to 3.0.6 so it should be relatively independend. :)
I can see this issue several times...

[nadouani]

Can you reproduce it by just running an analyzer or a given observable or does it occur only when you bulk run analyzers?

[crackytsi]

it seems to happen only during bulk Analysis (mark observable -> Action -> Analyse -> click more than one analyzer)

[crackytsi]

Problem still exists.
If I start an analyzer from the observable alone, reports are shown. If I use the bulk option, it does not work.

[nadouani]

This issue is really weird, working on my side

[crackytsi]

I added a new observable and started multiple analyzer for this selected observable.
I can see in the details all long reports as expected, but no short report is shown.
If I try to get all the observables of the case I see this for the observable I added:

     "tags" : [
         "demo"
      ],
      "_parent" : "AWKVDz4VufoPl6Jhh4eZ",
      "reports" : {
         "ANADNS_1_0" : {
            "taxonomies" : []
         }
      },
      "_routing" : "AWKVDz4VufoPl6Jhh4eZ",
      "updatedAt" : 1523537680133,
      "tlp" : 2,
      "id" : "c7af432217a2504c6f372adbaa822484",
      "updatedBy" : "usera",
      "ioc" : false,
      "status" : "Ok",
      "dataType" : "ip",
      "createdBy" : "usera",
      "_id" : "c7af432217a2504c6f372adbaa822484",
      "_type" : "case_artifact",
      "startDate" : 1523537660073,
      "message" : "",
      "createdAt" : 1523537660070,
      "_version" : 2,
      "data" : "130.122.14.195",
      "sighted" : false
   },

[nadouani]

I trust you :) but I don't know how to reproduce it :(

[crackytsi]

If I do the same manually I see this:

  {
      "tags" : [
         "demo"
      ],
      "message" : "",
      "createdBy" : "user",
      "tlp" : 2,
      "_routing" : "AWKVDz4VufoPl6Jhh4eZ",
      "_version" : 4,
      "data" : "130.122.14.195",
      "updatedAt" : 1523538152823,
      "_parent" : "AWKVDz4VufoPl6Jhh4eZ",
      "sighted" : false,
      "dataType" : "ip",
      "updatedBy" : "user",
      "status" : "Ok",
      "id" : "16314bc92809b55225f17a7141c95756",
      "createdAt" : 1523538115500,
      "reports" : {
         "ANADNS_1_0" : {
            "taxonomies" : [
               {
                  "predicate" : "Net",
                  "namespace" : "ANADNS",
                  "level" : "info",
                  "value" : "\"removed\""
               }
            ]
         },
         "OutManager_1_1" : {
            "taxonomies" : [
               {
                  "level" : "info",
                  "namespace" : "AM",
                  "value" : "\"removed\"",
                  "predicate" : "Server"
               }
            ]
         },
         "COMPONENT_1_0" : {
            "taxonomies" : [
               {
                  "predicate" : "Net",
                  "namespace" : "COMPONENT",
                  "level" : "info",
                  "value" : "\"removed\""
               }
            ]
         }
      },
      "_type" : "case_artifact",
      "ioc" : false,
      "startDate" : 1523538115506,
      "_id" : "16314bc92809b55225f17a7141c95756"
   },

[crackytsi]

Could this be related to the observable type?

[nadouani]

No the thing is that jobs are asynchronous and the issue here is that there is a conflict that ends by overriding the observable.reports property, which should have been fixed, but seems to appear again

[crackytsi]

It seems to be related if an analysis failes in any kind...

[nadouani]

you mean, if you run like 3 jobs and one of them fails?

[nadouani]

Hello,

what version of cortexutils do you have, and does your analyzer enable the "artifacts auto extraction"?

[crackytsi]

pip freeze | grep cortexutils
cortexutils==1.2.4
pip3 freeze | grep cortexutils
cortexutils==1.2.4

ExtractObservables=False

[nadouani]

ExtractObservables=False is the trick.

Update your enabled analyzers to make this option become true :)

[crackytsi]

Do you mean if this is disabled in Cortex-GUI but the analyzer results something TheHive behaves strange with the short-observable?

[nadouani]

Non, I'm just saying the artifacts = [] is a result of auto_extract_artifacts = False.

Nothing related to TheHive here. If you disable the extraction of artifacts by the analyzers, then the latters will not return a list of extracted observables

[nadouani]

Doe this occur with the "Official" analyzers? All the examples you provided are related to your "private" analyzers. Just curiosity :)

Any answer about this question (asked above few days ago)?

[crackytsi]

Strange, as this option is disabled for my analyzers as well as the others in the frontend.

[nadouani]

Why is this strange, I don't get you. it's disabled by default in the global configuration (just below the proxies)

[crackytsi]

I'm still trying to find the reason why sometimes the short-reports disappear or even don't appear if I run multiple analyzers as bulk instead of starting them one by one.
So this artifacts topic seemed to be different... But on my configuration side everything is configured the same (as I see...)

[nadouani]

Honestly, the artifacts thing is really not related to the short reports issue.

Please do you have an answer to my question above :)

[crackytsi]

It seems to be related only to self-developed ones.
I tested with 4 analyzers:

If I start it on case a) the short reports of analyzer 1 and 2 are shown.
If I start it on case b) with the same IP the short reports of analyzer 3 and 4 are shown.

So it seems to be randomly which result are shown/not shown.

[crackytsi]

I forgot to add that all use the same predicate but different namespaces.

If I do execute an api call api/case/artifact/_search:
Case a)

         "Analyzer_1" : {
            "taxonomies" : [
               {
                  "level" : "info",
                  "predicate" : "Net",
                  "value" : "\"somevaule\"",
                  "namespace" : "Analyzer1"
               }
            ]
         },
         "Analyzer_2" : {
            "taxonomies" : [
               {
                  "namespace" : "Analyzer2",
                  "level" : "info",
                  "value" : "\"someothervalue\"",
                  "predicate" : "Net"
               }
            ]
         }
      },

Case b)

      "reports" : {
         "Analyzer_3" : {
            "taxonomies" : [
               {
                  "namespace" : "Analyzer3",
                  "level" : "info",
                  "predicate" : "Server",
                  "value" : "\"somevalue\""
               }
            ]
         },
         "Analyzer_4" : {
            "taxonomies" : [
               {
                  "level" : "info",
                  "value" : "\"someothervalue\"",
                  "predicate" : "Net",
                  "namespace" : "Analyzer4"
               }
            ]
         }
      },

[crackytsi]

But if I click on the observable I can see that in both cases all 4 analyzers were executed and I seen see all 4 long reports.
On the observable view, I also see only 2 short-reports instead of 4.

So I don't thing that something on the template is wrong...

[nadouani]

If you do the same thing using 4 or 5 public analyzers, do you reproduce the issue?

[crackytsi]

Hi,
I can reproduce it also with public analyzers.
It seems to depend on the response time. If there is a big delay between the different reports, it seems not to occur.
You should be able to reproduce it if you have lets say 4 dummy analyzers that always bring directly the result. If you execute them for some the short reports will appear for others not. Randomly.

[crackytsi]

Btw: if I start an analysis again the logic could check if short Report / taxonomy is added and add it if missing.
Currently only long report gets added.
This would compensate the issue ;)

[nadouani]

This issue has been reproduced using 3 "fast" analyzers that just return the input

[rolinh]

Unfortunately, this issue is still not fixed for us (running 3.0.9 currently). We can reproduce it when running several (three is already enough) analyzers at once. Note that most of our analyzers are home-grown and return results quickly.

@crackytsi Can you confirm the fix or do you still experience this issue as well?

[nadouani]

@rolinh this is fixed in 3.0.10 that is not yet released

[nadouani]

it will come in the next few days

[rolinh]

@nadouani Ah, good to know, thanks. In which issue was this tracked? I missed it.

[nadouani]

In this one :)

We took 2 months to be able to reproduce and fine the reason, and it has been fixed in 30 minutes after reproducing it.

Most of the time has been spent on understanding that fast home made analyzers are required to reproduce the bug :)

[rolinh]

In this one :)

😁

It is not labeled 3.0.10 though 😉

[nadouani]

You're right, I was sure it had the correct milestone :(