MISP Pull takes forever and other performance issues.

[criptk]

System is 99% idle but MISP is ultra slow.
v2.4.136 is the version running on a Ubuntu 20.04.

Any pointers on how to troubleshoot performance on MISP ?

[mokaddem]

Hello,

Is it only for PULL or are you noticing the same slowness for other usage?

Could you also provide more information about

• The OS used
• The hardware (memory, cpu, sdd/hdd, ...)?
• The usage of the instance (#users, #correlations, #attributes & events, ...)
• How is the remote end performing (if the remote server is slow, it's normal that the PULL is taking time)?

Thanks

[criptk]

OS Used :
misp@MISP:$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
misp@MISP:$

The hardware (memory, cpu, sdd/hdd, ...):
2xCPU (it is not it, since it's idling 70%-99%)

The usage of the instance (#users, #correlations, #attributes & events, ...)
#users = 10
#correlations = 0 (disabled)
#attributes = 12095530
#events = 206062

How is the remote end performing (if the remote server is slow, it's normal that the PULL is taking time)?
it's running normally

It's not just the pull, some other pages take too long to respond (administration -> jobs, audit -> logs )

[criptk]

Sorry, it's running on a VM , SSD

[mokaddem]

How is the remote end performing (if the remote server is slow, it's normal that the PULL is taking time)?
it's running normally

Indeed, if the remote end is very slow, it will also slow down the pull process on your instance.

It's not just the pull, some other pages take too long to respond (administration -> jobs, audit -> logs )

However, these pages should not be slow as they are paginated. Just out of curiosity, how many entry do you have for your job and audit logs?

#correlations = 0 (disabled)
#attributes = 12095530
#events = 206062

The usage of the instance seems good. Around 50 attributes per events is a nice balance. However, please note that if you have lots of small events and some very large ones (containing +100000 attributes), it could explain the slowness of your instance.

May I ask how much memory does your VM provide as well as the memory_limit of PHP (a quick look at /servers/serverSettings/diagnostics does the job)?
It could also be interesting to have a look at the logs (/MISP/app/tmp/logs) to check if you don't have lots of exceptions being raised for some reasons.

Thanks!

[criptk]

memory_limit (2048 MB)… OK (recommended: 2048 MB)
The system has about 920 free.
Getting this error from resque-worker-error
--- snip ---
Error: Unknown named parameter $subject
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Event/CakeEventManager.php(244): ObjectCollection->trigger()
#1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3110): CakeEventManager->dispatch()
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(3020): Model->buildQuery()
#3 /var/www/MISP/app/Model/User.php(232): Model->find()
#4 [internal function]: User->__construct()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Utility/ClassRegistry.php(169): ReflectionClass->newInstance()
#6 /var/www/MISP/app/Plugin/SysLogLogable/Model/Behavior/SysLogLogableBehavior.php(269): ClassRegistry::init()
#7 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/BehaviorCollection.php(145): SysLogLogableBehavior->setup()
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/BehaviorCollection.php(66): BehaviorCollection->load()
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(781): BehaviorCollection->init()
#10 /var/www/MISP/app/Model/AppModel.php(56): Model->__construct()
#11 /var/www/MISP/app/Model/Event.php(278): AppModel->__construct()
#12 [internal function]: Event->__construct()
#13 /var/www/MISP/app/Lib/cakephp/lib/Cake/Utility/ClassRegistry.php(169): ReflectionClass->newInstance()
#14 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(316): ClassRegistry::init()
#15 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(269): Shell->loadModel()
#16 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(228): Shell->_loadModels()
#17 /var/www/MISP/app/Console/Command/AppShell.php(34): Shell->initialize()
#18 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#19 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#20 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform()
#21 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work()
#22 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker()
#23 {main}
--- snip ---