Zeek-Kafka not work , error in tests

[Canon88]

Summary of the issue

...
The plugin installation failed and the test could not be passed.

Expected behavior

...
Pass the test and complete the installation.

Steps to reproduce

...
$ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz
$ cd librdkafka-1.4.2/
$ ./configure --enable-sasl
$ make
$ sudo make install

$ zkg install seisollc/zeek-kafka

Logs, errors, etc.

...
$ zkg install seisollc/zeek-kafka
The following packages will be INSTALLED:
zeek/seisollc/zeek-kafka (v1.0.0)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
from zeek/seisollc/zeek-kafka (v1.0.0):
librdkafka ~1.4.2-RC1

Proceed? [Y/n] y
"zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root):
LIBRDKAFKA_ROOT: /usr/local
Saved answers to config file: /usr/local/zeek/etc/zkg/config
Running unit tests for "zeek/seisollc/zeek-kafka"
error: failed to run tests for zeek/seisollc/zeek-kafka: test_command failed with exit code 1
Proceed to install anyway? [N/y] n
Abort.

$ zeek -N Seiso::Kafka
error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: plugin Seiso::Kafka is not available
fatal error in /usr/local/zeek/share/zeek/base/init-bare.zeek, line 1: aborting after plugin errors

Your environment

• Version of Zeek：zeek version 4.2.0
• Version or commit hash of the zeek-kafka package
• Operating System and version：Ubuntu 20.04.4 LTS

[Canon88]

I see that the last updated support version is 4.0.5, is this because of the Zeek version?

[ottobackwards]

I have tested up to zeek 4.2.2 against main. I think we need to do a release, else you can try to install main instead of 1.0.0 in the mean time

[JonZeolla]

@Canon88 can you test against the latest version of zeek-kafka? We just did a v1.1.0-rc1 prerelease

[Canon88]

@Canon88 can you test against the latest version of zeek-kafka? We just did a v1.1.0-rc1 prerelease
No problem, on the way!

[Canon88]

@Canon88 can you test against the latest version of zeek-kafka? We just did a v1.1.0-rc1 prerelease

Can't specify the version to install, am I wrong somewhere? Please correct me?

$ zkg install seisollc/zeek-kafka --version 1.1.0-rc1
error: invalid package "seisollc/zeek-kafka": no such commit, branch, or version tag: "1.1.0-rc1"

The installation of 1.1.0-rc1 worked, but when I use the command to check the version, it is version 0.3.0.

$ zkg install seisollc/zeek-kafka
The following packages will be INSTALLED:
  zeek/seisollc/zeek-kafka (v1.1.0-rc1)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
  from zeek/seisollc/zeek-kafka (v1.1.0-rc1):
    librdkafka ~1.4.2

Proceed? [Y/n]
"zeek/seisollc/zeek-kafka" requires a "LIBRDKAFKA_ROOT" value (Path to librdkafka installation tree root):
LIBRDKAFKA_ROOT: /usr/local
Saved answers to config file: /usr/local/zeek/etc/zkg/config
Running unit tests for "zeek/seisollc/zeek-kafka"
Installing "zeek/seisollc/zeek-kafka"......................
Installed "zeek/seisollc/zeek-kafka" (v1.1.0-rc1)
Loaded "zeek/seisollc/zeek-kafka"

$ zeek -N Seiso::Kafka
Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)

[Canon88]

Can't specify the version to install, am I wrong somewhere? Please correct me?

$ zkg install seisollc/zeek-kafka --version 1.1.0-rc1
error: invalid package "seisollc/zeek-kafka": no such commit, branch, or version tag: "1.1.0-rc1"

[Canon88]

feedback

I set kafka.zeek, but it's not work. Kafka server address(192.168.199.98) is not valid, why connect 127.0.0.1:9092?

kafka.zeek

@load packages/zeek-kafka

redef Kafka::send_all_active_logs = T;

redef Kafka::tag_json = T;

#redef Kafka::logs_to_exclude = set(Conn::LOG);

redef Kafka::topic_name = "zeek";

redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "192.168.199.98:9092"
);

error.log

%3|1656089601.790|FAIL|rdkafka#producer-4| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|ERROR|rdkafka#producer-4| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|FAIL|rdkafka#producer-2| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|ERROR|rdkafka#producer-2| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|FAIL|rdkafka#producer-5| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|ERROR|rdkafka#producer-5| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.790|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.791|ERROR|rdkafka#producer-1| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.792|FAIL|rdkafka#producer-3| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.792|ERROR|rdkafka#producer-3| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.793|FAIL|rdkafka#producer-6| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1656089601.793|ERROR|rdkafka#producer-6| [thrd:127.0.0.1:9092/1001]: 127.0.0.1:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)

[ottobackwards]

you should not be changing kafka.zeek
you should be setting things in the local.zeek

[Canon88]

you should not be changing kafka.zeek you should be setting things in the local.zeek

yes, this my config. but i don't know why not work.

$ more kafka.zeek

@load packages/zeek-kafka

redef Kafka::send_all_active_logs = T;

redef Kafka::tag_json = T;

#redef Kafka::logs_to_exclude = set(Conn::LOG);

redef Kafka::topic_name = "zeek";

redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "192.168.199.98:9092"
);

$ more local.zeek

@load kafka

[Canon88]

I think I know what the problem is. This error is coming from my Kafka cluster. I use a Kafka cluster built by Docker. I tried to test it in a production environment and it worked fine.

It works fine in Zeek 4.2.2.