Kakfa plugin fails under FreeBSD 13.1

[clopmz]

Summary of the issue

Kafka install plugin works ok but when I run "zeekctl deploy" returns the following error:

==== stderr.log
error in /nsm/zeek/spool/installed-scripts-do-not-touch/site/packages/./zeek-kafka/./logs-to-kafka.zeek, line 25: unknown identifier logs_to_send, at or near "logs_to_send"
internal error in /nsm/zeek/spool/installed-scripts-do-not-touch/site/packages/./zeek-kafka/./logs-to-kafka.zeek, line 25: Failed to find variable named: Kafka::kafka_conf
/opt/zeek/share/zeekctl/scripts/run-zeek: line 110: 27422 Abort trap nohup "$myzeek" "$@"

Expected behavior

That it works

Steps to reproduce

• Install Zeek 4.0.6 under FreeBSD 13.1 with the following zkgs:

zeek/corelight/zeek-community-id (installed: 3.2.1) - "Community ID" flow hash support in conn.log
zeek/corelight/zeek-long-connections (installed: v1.2.0) - Find and log long-lived connections into a "conn_long" log.
zeek/salesforce/hassh (installed: master) - HASSH is used to identify specific Client and Server SSH implementations.
zeek/salesforce/ja3 (installed: master) - JA3 creates 32 character SSL client fingerprints and logs them as a field in ssl.log.
zeek/zeek/zeek-netmap (installed: v2.0.0) - Packet source plugin that provides native Netmap support.

• Install librdkafka-1.8.2 from FreeBSD ports and install zeek/seisollc/zeek-kafka using zkg.
• Configure Kafka plugin in Zeek with the following options:

redef Kafka::tag_json = T;
redef Kafka::send_all_active_logs = T;
redef Kafka::topic_name = "zeek";
redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "172.22.58.8:9092"
);

Where applicable, consider providing a patch that uses the end to end testing environment.

Logs, errors, etc.

==== stderr.log
error in /nsm/zeek/spool/installed-scripts-do-not-touch/site/packages/./zeek-kafka/./logs-to-kafka.zeek, line 25: unknown identifier logs_to_send, at or near "logs_to_send"
internal error in /nsm/zeek/spool/installed-scripts-do-not-touch/site/packages/./zeek-kafka/./logs-to-kafka.zeek, line 25: Failed to find variable named: Kafka::kafka_conf
/opt/zeek/share/zeekctl/scripts/run-zeek: line 110: 27422 Abort trap nohup "$myzeek" "$@"

Your environment

• Zeek 4.0.6
• zeek/seisollc/zeek-kafka (v1.0.0)
• FreeBSD 13.1

[ottobackwards]

If you just run zeek -NN does it work?

[clopmz]

Yes. Here it is:
......
Zeek::XMPP - XMPP analyzer (StartTLS only) (built-in)
[Analyzer] XMPP (ANALYZER_XMPP, enabled)
[Event] xmpp_starttls

Zeek::ZIP - Generic ZIP support analyzer (built-in)
[Analyzer] ZIP (enabled)

Corelight::CommunityID - "Community ID" flow hash support in the connection log (dynamic, version 3.2.0)
[Function] CommunityID::hash_conn

Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
[Writer] KafkaWriter (Log::WRITER_KAFKAWRITER)
[Constant] Kafka::kafka_conf
[Constant] Kafka::additional_message_values
[Constant] Kafka::topic_name
[Constant] Kafka::max_wait_on_shutdown
[Constant] Kafka::tag_json
[Constant] Kafka::json_timestamps
[Constant] Kafka::debug
[Constant] Kafka::mock
[Event] kafka_topic_resolved_event

Zeek::Netmap - Packet acquisition via Netmap (dynamic, version 1.0.0)
[Packet Source] NetmapReader (interface prefix "netmap"; supports live input)
[Packet Source] NetmapReader (interface prefix "vale"; supports live input)

Also installation goes well:
root@fbsdzeekmgmt:~ # zpkg install zeek/seisollc/zeek-kafka
The following packages will be INSTALLED:
zeek/seisollc/zeek-kafka (v1.0.0)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
from zeek/seisollc/zeek-kafka (v1.0.0):
librdkafka ~1.4.2-RC1

Proceed? [Y/n] y
zeek/seisollc/zeek-kafka asks for LIBRDKAFKA_ROOT (Path to librdkafka installation tree root) ? [/usr/local]
Saved answers to config file: /opt/zeek/etc/zkg/config
Running unit tests for "zeek/seisollc/zeek-kafka"
Installing "zeek/seisollc/zeek-kafka"..............................
Installed "zeek/seisollc/zeek-kafka" (v1.0.0)
Loaded "zeek/seisollc/zeek-kafka"

[ottobackwards]

I'm at a loss, I don't know, unless it is deployed incorrectly across the cluster I'm not sure why it would fail.
If you can zeek -NN on each of the cluster nodes, then it was deployed correctly. to all nodes.
I'm asking on Slack as you can see there

[clopmz]

Good morning,

One thing: "zeek -NN" only works on the manager but not in the workers:

root@fbsdnsm01:/opt/zeek/lib/zeek/plugins/packages # /opt/zeek/bin/zeek -NN
internal error: Failed to find variable named: Kafka::kafka_conf
Abort
root@fbsdnsm01:/opt/zeek/lib/zeek/plugins/packages #

[JonZeolla]

@clopmz it looks like you're running an old version of the package, can you attempt an update?

[clopmz]

Hi @JonZeolla ,

Uhmm ... older? Release installed in all zeek workers and manager are 1.8.2 ... According to https://github.com/edenhill/librdkafka/releases, latest release is 1.9.0 released 6 days ago ...

I will try it .... but I have serious doubts that this is it.

[clopmz]

Oops ... sorry ... My zeek-kafka package is release 1.0.0, and 1.1.0-rc1 was released 19 hours ago .....