New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Short Report is not shown on observables (3.0.8) #512

Closed
crackytsi opened this Issue Mar 22, 2018 · 53 comments

Comments

Projects
None yet
4 participants
@crackytsi

crackytsi commented Mar 22, 2018

Request Type

Bug

Work Environment

Question Answer
OS version (server) Debian
OS version (client) Seven using Chrome Browser
TheHive version / git hash 3.0.6
Package Type DEB

Problem Description

Unfortunately the short-report is still not always shown on observables (observables-tab and tab of the observable itself).

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

Can you provide the JSON definition of the observable? Just to see if it's a data issue or a display issue

@crackytsi

This comment has been minimized.

crackytsi commented Mar 23, 2018

How can I retrieve this definiton?

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

curl -XGET -H 'Authorization: Bearer API_KEY' 'http://SERVER:PORT/api/case/artifact/OBSERVABLE_ID'
@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

The OBSERVABLE_ID is the last part of the URL path of observable details page

@crackytsi

This comment has been minimized.

crackytsi commented Mar 23, 2018

{
   "_type" : "case_artifact",
   "updatedBy" : "user1",
   "createdAt" : 1520514455787,
   "createdBy" : "user1",
   "status" : "Ok",
   "data" : "192.168.200.2",
   "updatedAt" : 1520514472316,
   "_id" : "ea19684074eacd32ac5b1a087977fc30",
   "sighted" : false,
   "message" : "environment1",
   "reports" : {
      "HASDB_1_0" : {
         "taxonomies" : [
            {
               "predicate" : "Net",
               "level" : "info",
               "value" : "\"192.168.200.0/28 NPS FCARAS123\"",
               "namespace" : "HASDB"
            }
         ]
      }
   },
   "ioc" : false,
   "tlp" : 2,
   "dataType" : "ip",
   "id" : "ea19684074eacd32ac5b1a087977fc30",
   "tags" : [
      "environment1"
   ],
   "startDate" : 1520514455790
}

@crackytsi

This comment has been minimized.

crackytsi commented Mar 23, 2018

Hi,
As you might have noticed, this are analyzers that we use internal.
Strangely on the overview table, I can see Reports for 2 analyzers (HASDB and AssetDB), both Long reports are shown correctly.

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

I'm taking a look :)

@nadouani nadouani added the bug label Mar 23, 2018

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

Well all looks good, the data (reports attribute) is on the observable, well formed too.

Do you have any javascript error on your browsers console?

@crackytsi

This comment has been minimized.

crackytsi commented Mar 23, 2018

There are 2 reports.... not just 1.

The problem seems to be that not always all reports are added to the data structure.
No, I have no javascript error.

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

Wait, wait. Your observables should have 2 reports but it's definition contains only one instead of 2?

In the observable details page, you see both mini reports, and in observables list you see only one mini report?

Just to be sure we are talking about the same thing

@crackytsi

This comment has been minimized.

crackytsi commented Mar 23, 2018

Hi,
Right, my observable should have 2 reports, but the Definition retrieved by the API Call listes only one.

In the observable page I see 2 reports, but there is also only one mini-reports.
If I click on the analyzer, I see both Long reports.

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

and both analyzer jobs are successful right?

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

This sounds like #409

@crackytsi

This comment has been minimized.

crackytsi commented Mar 23, 2018

Both Jobs were successfull.
yes it does, but this issue was solved and I'm using version 3.0.6 :-/

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

Yes, I know, but it could has been solved partially, or you could have a corner case that has not been taken into account. The problem with this issue is that it's not easy to reproduce.

@To-om will take a look on it

@crackytsi

This comment has been minimized.

crackytsi commented Mar 23, 2018

If I can support you, let me know
The Case was created after upgrade to 3.0.6 so it should be relatively independend. :)
I can see this issue several times...

@nadouani

This comment has been minimized.

Contributor

nadouani commented Mar 23, 2018

Can you reproduce it by just running an analyzer or a given observable or does it occur only when you bulk run analyzers?

@crackytsi

This comment has been minimized.

crackytsi commented Mar 23, 2018

it seems to happen only during bulk Analysis (mark observable -> Action -> Analyse -> click more than one analyzer)

@crackytsi crackytsi changed the title from Short Report is not shown on observables (3.0.6) to Short Report is not shown on observables (3.0.8) Apr 12, 2018

@crackytsi

This comment has been minimized.

crackytsi commented Apr 12, 2018

Problem still exists.
If I start an analyzer from the observable alone, reports are shown. If I use the bulk option, it does not work.

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 12, 2018

This issue is really weird, working on my side

dashboard

@crackytsi

This comment has been minimized.

crackytsi commented Apr 12, 2018

I added a new observable and started multiple analyzer for this selected observable.
I can see in the details all long reports as expected, but no short report is shown.
If I try to get all the observables of the case I see this for the observable I added:

     "tags" : [
         "demo"
      ],
      "_parent" : "AWKVDz4VufoPl6Jhh4eZ",
      "reports" : {
         "ANADNS_1_0" : {
            "taxonomies" : []
         }
      },
      "_routing" : "AWKVDz4VufoPl6Jhh4eZ",
      "updatedAt" : 1523537680133,
      "tlp" : 2,
      "id" : "c7af432217a2504c6f372adbaa822484",
      "updatedBy" : "usera",
      "ioc" : false,
      "status" : "Ok",
      "dataType" : "ip",
      "createdBy" : "usera",
      "_id" : "c7af432217a2504c6f372adbaa822484",
      "_type" : "case_artifact",
      "startDate" : 1523537660073,
      "message" : "",
      "createdAt" : 1523537660070,
      "_version" : 2,
      "data" : "130.122.14.195",
      "sighted" : false
   },

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 12, 2018

I trust you :) but I don't know how to reproduce it :(

@crackytsi

This comment has been minimized.

crackytsi commented Apr 12, 2018

If I do the same manually I see this:

  {
      "tags" : [
         "demo"
      ],
      "message" : "",
      "createdBy" : "user",
      "tlp" : 2,
      "_routing" : "AWKVDz4VufoPl6Jhh4eZ",
      "_version" : 4,
      "data" : "130.122.14.195",
      "updatedAt" : 1523538152823,
      "_parent" : "AWKVDz4VufoPl6Jhh4eZ",
      "sighted" : false,
      "dataType" : "ip",
      "updatedBy" : "user",
      "status" : "Ok",
      "id" : "16314bc92809b55225f17a7141c95756",
      "createdAt" : 1523538115500,
      "reports" : {
         "ANADNS_1_0" : {
            "taxonomies" : [
               {
                  "predicate" : "Net",
                  "namespace" : "ANADNS",
                  "level" : "info",
                  "value" : "\"removed\""
               }
            ]
         },
         "OutManager_1_1" : {
            "taxonomies" : [
               {
                  "level" : "info",
                  "namespace" : "AM",
                  "value" : "\"removed\"",
                  "predicate" : "Server"
               }
            ]
         },
         "COMPONENT_1_0" : {
            "taxonomies" : [
               {
                  "predicate" : "Net",
                  "namespace" : "COMPONENT",
                  "level" : "info",
                  "value" : "\"removed\""
               }
            ]
         }
      },
      "_type" : "case_artifact",
      "ioc" : false,
      "startDate" : 1523538115506,
      "_id" : "16314bc92809b55225f17a7141c95756"
   },

@crackytsi

This comment has been minimized.

crackytsi commented Apr 12, 2018

Could this be related to the observable type?

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 12, 2018

No the thing is that jobs are asynchronous and the issue here is that there is a conflict that ends by overriding the observable.reports property, which should have been fixed, but seems to appear again

@crackytsi

This comment has been minimized.

crackytsi commented Apr 13, 2018

It seems to be related if an analysis failes in any kind...

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 13, 2018

you mean, if you run like 3 jobs and one of them fails?

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 17, 2018

Hello,

what version of cortexutils do you have, and does your analyzer enable the "artifacts auto extraction"?

@crackytsi

This comment has been minimized.

crackytsi commented Apr 17, 2018

pip freeze | grep cortexutils
cortexutils==1.2.4
pip3 freeze | grep cortexutils
cortexutils==1.2.4

ExtractObservables=False

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 17, 2018

ExtractObservables=False is the trick.

Update your enabled analyzers to make this option become true :)

@crackytsi

This comment has been minimized.

crackytsi commented Apr 17, 2018

Do you mean if this is disabled in Cortex-GUI but the analyzer results something TheHive behaves strange with the short-observable?

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 17, 2018

Non, I'm just saying the artifacts = [] is a result of auto_extract_artifacts = False.

Nothing related to TheHive here. If you disable the extraction of artifacts by the analyzers, then the latters will not return a list of extracted observables

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 17, 2018

Doe this occur with the "Official" analyzers? All the examples you provided are related to your "private" analyzers. Just curiosity :)

Any answer about this question (asked above few days ago)?

@crackytsi

This comment has been minimized.

crackytsi commented Apr 17, 2018

Strange, as this option is disabled for my analyzers as well as the others in the frontend.

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 17, 2018

Why is this strange, I don't get you. it's disabled by default in the global configuration (just below the proxies)

@crackytsi

This comment has been minimized.

crackytsi commented Apr 17, 2018

I'm still trying to find the reason why sometimes the short-reports disappear or even don't appear if I run multiple analyzers as bulk instead of starting them one by one.
So this artifacts topic seemed to be different... But on my configuration side everything is configured the same (as I see...)

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 17, 2018

Honestly, the artifacts thing is really not related to the short reports issue.

Please do you have an answer to my question above :)

@crackytsi

This comment has been minimized.

crackytsi commented Apr 17, 2018

It seems to be related only to self-developed ones.
I tested with 4 analyzers:

If I start it on case a) the short reports of analyzer 1 and 2 are shown.
If I start it on case b) with the same IP the short reports of analyzer 3 and 4 are shown.

So it seems to be randomly which result are shown/not shown.

@crackytsi

This comment has been minimized.

crackytsi commented Apr 17, 2018

I forgot to add that all use the same predicate but different namespaces.

If I do execute an api call api/case/artifact/_search:
Case a)

         "Analyzer_1" : {
            "taxonomies" : [
               {
                  "level" : "info",
                  "predicate" : "Net",
                  "value" : "\"somevaule\"",
                  "namespace" : "Analyzer1"
               }
            ]
         },
         "Analyzer_2" : {
            "taxonomies" : [
               {
                  "namespace" : "Analyzer2",
                  "level" : "info",
                  "value" : "\"someothervalue\"",
                  "predicate" : "Net"
               }
            ]
         }
      },

Case b)

      "reports" : {
         "Analyzer_3" : {
            "taxonomies" : [
               {
                  "namespace" : "Analyzer3",
                  "level" : "info",
                  "predicate" : "Server",
                  "value" : "\"somevalue\""
               }
            ]
         },
         "Analyzer_4" : {
            "taxonomies" : [
               {
                  "level" : "info",
                  "value" : "\"someothervalue\"",
                  "predicate" : "Net",
                  "namespace" : "Analyzer4"
               }
            ]
         }
      },
@crackytsi

This comment has been minimized.

crackytsi commented Apr 17, 2018

But if I click on the observable I can see that in both cases all 4 analyzers were executed and I seen see all 4 long reports.
On the observable view, I also see only 2 short-reports instead of 4.

So I don't thing that something on the template is wrong...

@nadouani

This comment has been minimized.

Contributor

nadouani commented Apr 17, 2018

If you do the same thing using 4 or 5 public analyzers, do you reproduce the issue?

@crackytsi

This comment has been minimized.

crackytsi commented Apr 18, 2018

Hi,
I can reproduce it also with public analyzers.
It seems to depend on the response time. If there is a big delay between the different reports, it seems not to occur.
You should be able to reproduce it if you have lets say 4 dummy analyzers that always bring directly the result. If you execute them for some the short reports will appear for others not. Randomly.

@crackytsi

This comment has been minimized.

crackytsi commented Apr 20, 2018

Btw: if I start an analysis again the logic could check if short Report / taxonomy is added and add it if missing.
Currently only long report gets added.
This would compensate the issue ;)

@nadouani

This comment has been minimized.

Contributor

nadouani commented May 3, 2018

This issue has been reproduced using 3 "fast" analyzers that just return the input

@Rolinh

This comment has been minimized.

Rolinh commented May 22, 2018

Unfortunately, this issue is still not fixed for us (running 3.0.9 currently). We can reproduce it when running several (three is already enough) analyzers at once. Note that most of our analyzers are home-grown and return results quickly.

@crackytsi Can you confirm the fix or do you still experience this issue as well?

@nadouani

This comment has been minimized.

Contributor

nadouani commented May 22, 2018

@Rolinh this is fixed in 3.0.10 that is not yet released

@nadouani

This comment has been minimized.

Contributor

nadouani commented May 22, 2018

it will come in the next few days

@Rolinh

This comment has been minimized.

Rolinh commented May 22, 2018

@nadouani Ah, good to know, thanks. In which issue was this tracked? I missed it.

@nadouani

This comment has been minimized.

Contributor

nadouani commented May 22, 2018

In this one :)

We took 2 months to be able to reproduce and fine the reason, and it has been fixed in 30 minutes after reproducing it.

Most of the time has been spent on understanding that fast home made analyzers are required to reproduce the bug :)

@Rolinh

This comment has been minimized.

Rolinh commented May 23, 2018

In this one :)

😁

It is not labeled 3.0.10 though 😉

@nadouani

This comment has been minimized.

Contributor

nadouani commented May 23, 2018

You're right, I was sure it had the correct milestone :(

@nadouani nadouani added this to the 3.0.10 milestone May 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment