Zeek-Kafka not installed properly , error in tests

[cybersecurity99]

Summary of the issue

...
I tried to install zeek-kafka but it failed

Expected behavior

...
It should get installed and work

Steps to reproduce

...
installed librdkafka and libsasl2-dev

Then
$ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz
$ cd librdkafka-1.4.2/
$ ./configure --enable-sasl
$ make
$ sudo make install
Where applicable, consider providing a patch that uses the end to end testing environment.

Logs, errors, etc.

...
root@admin:/usr/local/zeek/bin# ./zkg install seisollc/zeek-kafka
The following packages will be INSTALLED:
zeek/seisollc/zeek-kafka (v1.0.0)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
from zeek/seisollc/zeek-kafka (v1.0.0):
librdkafka ~1.4.2-RC1

Proceed? [Y/n] y
"zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root):
LIBRDKAFKA_ROOT: /usr/local
Saved answers to config file: /usr/local/zeek/etc/zkg/config
Running unit tests for "zeek/seisollc/zeek-kafka"
error: failed to run tests for zeek/seisollc/zeek-kafka: test_command failed with exit code 1
Proceed to install anyway? [N/y]

Your environment

• Version of Zeek 4.0.5
• Version or commit hash of the zeek-kafka package main
• Operating System and version ubuntu 20.04

[JonZeolla]

@cybersecurity99 Hmm I ran our end to end tests on Ubuntu 20.04 with Zeek 4.0.5 and it seems to be working. Are you seeing anything substantially different between your environment and what we are doing in Docker?

#38

[cybersecurity99]

@cybersecurity99 Hmm I ran our end to end tests on Ubuntu 20.04 with Zeek 4.0.5 and it seems to be working. Are you seeing anything substantially different between your environment and what we are doing in Docker?

#38

@JonZeolla I am not using Docker , I have tried this in Multiple VM . I am facing this in all attempts also if I install after failed test I get this when checking installation

./zeek -N Seiso::Kafka
error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: cannot load plugin library /usr/local/zeek/lib/zeek/plugins/packages/zeek-kafka//lib/SEISO-KAFKA.linux-x86_64.so: /usr/local/lib/librdkafka++.so.1: undefined symbol: rd_kafka_error_string
fatal error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

Is this any configuration file or logs which I can check to see real reason for error

[JonZeolla]

@cybersecurity99 are you using the right version of librdkafka? We only currently support 1.4.2 or 1.4.4

[cybersecurity99]

@cybersecurity99 are you using the right version of librdkafka? We only currently support 1.4.2 or 1.4.4

@JonZeolla , yes I guess I downloaded from apt install librdkafka-dev.
Also
I tried to delete zeek-kafka packages and reinstall it
It installed it without any error . But it doesn't show it ran 10 test cases successfully or failed it just showed running test cases and then installed .
It is sending data to kafka But at the same time it is writing logs in zeek/logs/current folder how to stop that or is it permanent ?

[JonZeolla]

@cybersecurity99 at a quick glance that may be installing the wrong version of librdkafka. May be looking at the wrong thing via my phone.

https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/librdkafka-dev_1.2.1-1ubuntu1_arm64.deb.html

You could use a modified version of https://github.com/JonZeolla/Development/blob/76d4bc84e222544baec9e8269d71f3cfa55c0869/python/librdkafka_version_checker.py to check the version you have on disk, or more simply you could just use apt/dpkg.

I'm not sure why this package would create different logs on disk than you already had... @ottobackwards any idea on that one

[cybersecurity99]

@cybersecurity99 at a quick glance that may be installing the wrong version of librdkafka. May be looking at the wrong thing via my phone.

https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/librdkafka-dev_1.2.1-1ubuntu1_arm64.deb.html

You could use a modified version of https://github.com/JonZeolla/Development/blob/76d4bc84e222544baec9e8269d71f3cfa55c0869/python/librdkafka_version_checker.py to check the version you have on disk, or more simply you could just use apt/dpkg.

I'm not sure why this package would create different logs on disk than you already had... @ottobackwards any idea on that one

Hi @JonZeolla
I tried that python script to check but got some error , is there any pre requisite to run that script

from: can't read /var/mail/ctypes
./librdkafka_version_checker.py: line 2: minver: command not found
./librdkafka_version_checker.py: line 3: syntax error near unexpected token `('
./librdkafka_version_checker.py: line 3: `dll = cdll.LoadLibrary("/usr/local/lib/librdkafka.so")'
 

Also you pointed out that 1.4.2 and 1.4.4 is supported so , we don't need to download librdkafka from apt ? Just that tar file and steps mentioned in 1st point are enough to run.
Because I have done apt install librdkafka , its shows it to be 1.8.2 .
So should I just remove them and use

$ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz
$ cd librdkafka-1.4.2/
$ ./configure --enable-sasl
$ make
$ sudo make install

After doing so and reinstalling zeek-kafka I found that data was sent to kafka and only 1 log was generated in /logs/current/ folder that is .crash-diag.out which contained following info

root@zeek-VirtualBox:/usr/local/zeek/logs/current# cat .crash-diag.out 
 
Zeek 5.0.0-dev.280
Linux 5.13.0-40-generic
 
Zeek plugins:
Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
 
==== No reporter.log
 
==== No stderr.log
 
==== No stdout.log
 
==== No .cmdline
 
==== No .env_vars
 
==== No .status
 
==== No prof.log
 
==== No packet_filter.log
 
==== No loaded_scripts.log

What it shows ?

Also how to do we differentiate data in zeek kafka topic like http.log , conn.log ,etc

Edit
When I tried by commenting zeek-kafka in local.zeek , there were no logs in current folder also
Thanks

[JonZeolla]

@cybersecurity99 wait you're running Zeek 5? That's definitely not supported yet. Also, correct librdkafka 1.8.2 isn't supported

If updating your versions fixes things please close this issue, otherwise let me know and we can continue to troubleshoot.

[cybersecurity99]

@cybersecurity99 wait you're running Zeek 5? That's definitely not supported yet. Also, correct librdkafka 1.8.2 isn't supported

If updating your versions fixes things please close this issue, otherwise let me know and we can continue to troubleshoot.

@JonZeolla
I took this in consideration and created a fresh machine with zeek v4.0.5 , and librdkafka 1.4.2 as given in commands

Still same issue it is sending data to kafka
But at the same time writing logs in /zeek/logs/current
Can you check this does it write for you also in that folder ?

[JonZeolla]

@cybersecurity99 just to be clear, it is sending logs to Kafka like you want, but it is also storing them on disk and you want to disable that? This project adds the ability to send logs to Kafka, it doesn't replace any local log writing, but you can disable that separately.

I suggest you check out https://docs.zeek.org/en/master/frameworks/logging.html#terminology and go from there

[cybersecurity99]

@cybersecurity99 just to be clear, it is sending logs to Kafka like you want, but it is also storing them on disk and you want to disable that? This project adds the ability to send logs to Kafka, it doesn't replace any local log writing, but you can disable that separately.

I suggest you check out https://docs.zeek.org/en/master/frameworks/logging.html#terminology and go from there

@JonZeolla okay got that part.

Just wanna know if send all logs together. How do we differentiate between http.log,conn.log ,etc at kafka end .
Any command to print specific log type using topic "zeek"

[JonZeolla]

@cybersecurity99 I think you're looking for something like tag_json and then post processing the logs after receiving them from Kafka. You could also send different logs to different topics

https://github.com/SeisoLLC/zeek-kafka#tag_json