Support: MISP Could not fetch feed

[patchander]

Support Questions

Hello I have an issue with my MISP instance pulling a particular feed. I have searched GitHub and found a similar issue: #814

Currently I have a feed in MISP which has initially pulled events but is now reporting errors each time I pull events and shows "Could not fetch feed" within jobs. Please find attached screenshot from the Jobs page and errors below.

I have tried the following troubleshooting steps:

• Checked each user has an organisation assigned to it.
• Checked the database, org_id is present in tables: events, jobs, servers, shadow_attributes, threads, users and correlations.
• There are no blank org_id fields within my database for any table.
• Spoke to the vendor about the data and they have confirmed their data is ok.
• Based on Column 'org_id' cannot be null #814 - this issue asks to check the database for org. I cannot find any column with org so I am assuming this advice is for an older version of MISP.

Please can you advise on steps to resolve my issue?

For info my MISP was installed from https://github.com/amuehlem/MISP-RPM

Thanks

MISP version

2.4.150

Operating System

RedHat

Operating System version

7.9

PHP version

7.4.25

Browser

Chrome

Browser version

99.0.4844.51

Relevant log output

Error: Could not edit event 'e48dfc04-0752-4c6c-807a-99d9a0431726' from feed 27.
[PDOException] SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'org_id' cannot be null
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/Database/Mysql.php(914): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Correlation.php(279): Mysql->insertMulti()
#2 /var/www/MISP/app/Model/Correlation.php(385): Correlation->__saveCorrelations()
#3 /var/www/MISP/app/Model/Attribute.php(442): Correlation->afterSaveCorrelation()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Event/CakeEventManager.php(242): Attribute->afterSave()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1970): CakeEventManager->dispatch()
#6 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave()
#7 /var/www/MISP/app/Model/Attribute.php(3726): Model->save()
#8 /var/www/MISP/app/Model/Event.php(4075): Attribute->editAttribute()
#9 /var/www/MISP/app/Model/Feed.php(1031): Event->_edit()
#10 /var/www/MISP/app/Model/Feed.php(706): Feed->__updateEventFromFeed()
#11 /var/www/MISP/app/Model/Feed.php(1083): Feed->downloadFromFeed()
#12 /var/www/MISP/app/Console/Command/ServerShell.php(467): Feed->downloadFromFeedInitiator()
#13 /var/www/MISP/app/Console/Command/AppShell.php(36): ServerShell->enqueueFeedFetch()
#14 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#15 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#16 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform()
#17 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work()
#18 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker()

Extra attachments

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[righel]

Hello @patchander
Could you please update MISP to the latest version and check if the error persists?
Could you try enabling MISP.deadlock_avoidance and try to pull the feed again?

Let me know how that goes.

[patchander]

Hello, I made the suggested change to the MISP.deadlock_avoidance setting however I still have the same issue.

Below is the error message in the error.log:

2022-05-11 10:47:02 Error: Could not edit event '1bded382-422c-4675-9e1f-fd216f9fc2a2' from feed 27.
[PDOException] SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'org_id' cannot be null
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/Database/Mysql.php(914): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Correlation.php(279): Mysql->insertMulti()
#2 /var/www/MISP/app/Model/Correlation.php(385): Correlation->__saveCorrelations()
#3 /var/www/MISP/app/Model/Attribute.php(442): Correlation->afterSaveCorrelation()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Event/CakeEventManager.php(242): Attribute->afterSave()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1970): CakeEventManager->dispatch()
#6 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave()
#7 /var/www/MISP/app/Model/Attribute.php(3726): Model->save()
#8 /var/www/MISP/app/Model/Event.php(4075): Attribute->editAttribute()
#9 /var/www/MISP/app/Model/Feed.php(1031): Event->_edit()
#10 /var/www/MISP/app/Model/Feed.php(706): Feed->__updateEventFromFeed()
#11 /var/www/MISP/app/Model/Feed.php(1083): Feed->downloadFromFeed()
#12 /var/www/MISP/app/Console/Command/ServerShell.php(255): Feed->downloadFromFeedInitiator()
#13 /var/www/MISP/app/Console/Command/AppShell.php(36): ServerShell->fetchFeed()
#14 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#15 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#16 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform()
#17 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work()
#18 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker()
#19 {main}
2022-05-11 10:47:10 Error: Could not edit event 'a3e7ff26-7ce6-456b-90ec-0bb61166bf2b' from feed 27.
[PDOException] SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'org_id' cannot be null
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/Database/Mysql.php(914): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Correlation.php(279): Mysql->insertMulti()
#2 /var/www/MISP/app/Model/Correlation.php(385): Correlation->__saveCorrelations()
#3 /var/www/MISP/app/Model/Attribute.php(442): Correlation->afterSaveCorrelation()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Event/CakeEventManager.php(242): Attribute->afterSave()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1970): CakeEventManager->dispatch()
#6 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave()
#7 /var/www/MISP/app/Model/Attribute.php(3726): Model->save()
#8 /var/www/MISP/app/Model/Event.php(4075): Attribute->editAttribute()
#9 /var/www/MISP/app/Model/Feed.php(1031): Event->_edit()
#10 /var/www/MISP/app/Model/Feed.php(706): Feed->__updateEventFromFeed()
#11 /var/www/MISP/app/Model/Feed.php(1083): Feed->downloadFromFeed()
#12 /var/www/MISP/app/Console/Command/ServerShell.php(255): Feed->downloadFromFeedInitiator()
#13 /var/www/MISP/app/Console/Command/AppShell.php(36): ServerShell->fetchFeed()
#14 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#15 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#16 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform()
#17 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work()
#18 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker()
#19 {main}
2022-05-11 10:47:13 Error: Could not edit event 'c9a65877-d166-408d-8eab-b05e6f04888b' from feed 27.
[PDOException] SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'org_id' cannot be null
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/Database/Mysql.php(914): PDOStatement->execute()
#1 /var/www/MISP/app/Model/Correlation.php(279): Mysql->insertMulti()
#2 /var/www/MISP/app/Model/Correlation.php(385): Correlation->__saveCorrelations()
#3 /var/www/MISP/app/Model/Attribute.php(442): Correlation->afterSaveCorrelation()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Event/CakeEventManager.php(242): Attribute->afterSave()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1970): CakeEventManager->dispatch()
#6 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave()
#7 /var/www/MISP/app/Model/Attribute.php(3726): Model->save()
#8 /var/www/MISP/app/Model/Event.php(4075): Attribute->editAttribute()
#9 /var/www/MISP/app/Model/Feed.php(1031): Event->_edit()
#10 /var/www/MISP/app/Model/Feed.php(706): Feed->__updateEventFromFeed()
#11 /var/www/MISP/app/Model/Feed.php(1083): Feed->downloadFromFeed()
#12 /var/www/MISP/app/Console/Command/ServerShell.php(255): Feed->downloadFromFeedInitiator()
#13 /var/www/MISP/app/Console/Command/AppShell.php(36): ServerShell->fetchFeed()
#14 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#15 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#16 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform()
#17 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work()
#18 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker()
#19 {main}

Please can you help?