MISP can't fetch events from feed with Kafka enabled

[freshm1nce]

MISP can't produce messages to kafka's topic when fetching feeds. But if manually add attributes, all works fine.

Work environment

Questions	Answers
Type of issue	Bug
OS version (server)	Ubuntu 20.04.1 LTS x86_64
OS version (client)	Win10 x64
PHP version	7.4
MISP version / git hash	v2.4.135 (ce96003)
Browser	Chrome

Expected behavior

Misp adding events with attributes and creates kafka messages.

Actual behavior

Misp creates 1 event with 0 attributes. fetch_feed job failed.

Steps to reproduce the behavior

install misp on clear ubuntu server by instruction here:https://misp.github.io/MISP/INSTALL.ubuntu2004/.
Then instal kafka module with also that instruction(plus add in /etc/php/7.4/apache2/php.ini and plus add in /etc/php/7.4/cli/php.ini "extension=rdkafka.so"). Configure module Kafka in web interface. Enable some feed, enable caching. Start fetching events in any way.

Logs, screenshots, configuration dump, ...

I saw errors in resque-worker-error.log "Error: Class 'RdKafka\Conf' not found
#0 /var/www/MISP/app/Model/AppModel.php(2506): KafkaPubTool->initTool()
#1 /var/www/MISP/app/Model/AppModel.php(2491): AppModel->loadKafkaPubTool()
#2 /var/www/MISP/app/Model/Attribute.php(786): AppModel->getKafkaPubTool()
#3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Event/CakeEventManager.php(242): Attribute->afterSave()
#4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1970): CakeEventManager->dispatch()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave()
#6 /var/www/MISP/app/Model/Attribute.php(4216): Model->save()
#7 /var/www/MISP/app/Model/Event.php(3961): Attribute->captureAttribute()
#8 /var/www/MISP/app/Model/Feed.php(862): Event->_add()
#9 /var/www/MISP/app/Model/Feed.php(551): Feed->__addEventFromFeed()
#10 /var/www/MISP/app/Model/Feed.php(935): Feed->downloadFromFeed()
#11 /var/www/MISP/app/Console/Command/ServerShell.php(250): Feed->downloadFromFeedInitiator()
#12 /var/www/MISP/app/Console/Command/AppShell.php(35): ServerShell->fetchFeed()
#13 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#14 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#15 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform()
#16 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work()
#17 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker()
#18 {main}
"
but when i manually adding some attributes to events, misp is working fine and produces messages to kafka's topic.. So it's one place (maybe not) when he can't do it. Could it be access rights problem with worker? I think that also because of php 7.4 version.

Logs and config.php in attach to this issue.
logs + config.zip

[JakubOnderka]

The problem will be that the extension is enable for web interface, but not for CLI. Maybe you use different PHP version? Can you check if rdkafka is really install also for CLI by running php -m command?

[freshm1nce]

@JakubOnderka I installed rdkafka + wrote "extension=rdkafka.so" into both files: /etc/php/7.4/cli/php.ini and /etc/php/7.4/apache2/php.ini. By running php -m command, i see this result:
[PHP Modules]
calendar
Core
ctype
date
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
gnupg
hash
iconv
igbinary
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
Phar
posix
rdkafka
readline
redis
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
and php -i:
:/var/www/MISP$ php -i | grep kafka
rdkafka
rdkafka support => enabled
librdkafka version (runtime) => 1.2.1
librdkafka version (build) => 1.2.1.255

I also tried to understand How it's working. MISP just uses /var/www/MISP/app/Lib/Tools/KafkaPubTool.php which is Kafka's producer. All actions with kafka are the same in MISP, they just use KafkaPubTool\InitTool(params), so i don't understand, why it is working with manually adding attributes via web interface, but doesn't with worker's task "fetch feed events".
Seems like worker can't access/use KafkapubTool.php

In older versions there is no issues with it. But not in 2.4.135 with php 7.2 ..

[freshm1nce]

I can work around the bug by killing the workers and restart them once. Does anyone have any suggestion what what is causing this issue?