[Bug] Slow loading of TheHive because of Tags

[crackytsi]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian
OS version (client)	10
TheHive version / git hash	4.1.0
Package Type	DEB

Problem Description

Starting the TheHive page or directly jumping to a case using a link takes several times.
By analyzing this, the root-cause seems to be related to URL:

/api/v1/query?name=list-tags The Tag-List on my test system with 800 Cases and 7366 Tags is about 2 MB. That means on my prod system the tag-list with arround 20000 Cases is more than 20x slower

Unfortunately this prevents me from migrating to 4.1 in Prod :-(

Possible Solutions

Do not provide Tag-list to FE and auto-complete Tag list based on backend.
It seems that this functionality is only used for auto-complete.

Alternative specify which tags should be auto-completed (e.g. all non-free tags).

BTW: Why are there 2 ~chars in the URL of a case? /index.html#!/case/~~44187888/details

Solution

The solution is to cache only the free tags that define a custom colour:

• BE (@To-om) Add free tag custom colour to /api/status output
• FE (@nadouani) Cache only free tags with custom colour

[nadouani]

The issue here is that you have 7k+ tags. I'm pretty sure you have "data" in your tags, like emails, ids, etc.. but well this comes from how TheHive 3 is used.

That means on my prod system the tag-list with around 20000 Cases is more than 20x slower

I'm not sure this assumption is true, except if each case comes with 10 new/distinct tags (which is not the right way of doing things).

Anyways, we will think of how to improve this but we need the tags data if we want to display the colours...

[crackytsi]

@nadouani Thank you.
Actually there is only one tag on case level (Offense-ID) that should be moved to a custom field (in TH3 there was no other option to filter on that attribute)
On observable-level I have a lot of tags that contain the alarm-id of the observable. Thats necessary to be able to get the "link" between an observable and the related Alert. So far I don't see any real alternative for doing so, because from operation perspective its extremly helpfull to see on the observables which attributes they related to (in the SIEM System). Keep in mind, that custom-fields doesn't exist on observable-level and doesn't support multi-value attributes.

Attached a screenshot with some examples:

[nadouani]

OK I get it for the observables. I get if for TheHive 3 and custom fields.

Now with 20k cases => 20k Alert Ids as tags. A I support, thousands of alerts and observables with their tags. This is in fact a use case we didn't though about. We have to brainstorm here to come up with a solution

[nadouani]

Found a solution: Have the free tags cache API only load the free tags that defined a custom colour instead of loading all the tags ;)

[crackytsi]

@nadouani Thanks, sounds great and should work :) Does this affect Auto-complete?

Another idear came in my mind this night, so I just wanted to share it with you:
TH could have something like "dynamic-free-tags" classified by namespace and prediacted.
e.g. all tags with (old) prefix "alarmid=" can be ignored, if alarmid is added to a list of "dynamic free tags".

[nadouani]

No, auto complete relies on an API

[crackytsi]

@nadouani @To-om With 4.1.2 Tags still load very slow, in my case it takes for 900 17 seconds: