Error while running

[cybersecurity99]

11/4/2022 -- 12:44:36 - - brokers parameter required!
thread '' panicked at 'explicit panic', src/lib.rs:55:13
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Aborted (core dumped)

[Center-Sun]

Hi, did you add kafka section to your suricata.yaml? seen plugin can't find some necessary parameters.

[cybersecurity99]

Hi @Center-Sun , added this at end of file , exact copy paste in suricata.yaml

kafka:
  brokers: "kafka1:9092,kafka2:9092"
  topic: suricata
  client-id: suricata_client_01
  buffer-size: 1024

I replaced kafka:port with my own server details. plugin is detected by suricata but it can't read parameters . Request you to try a demo of it in a Virtual Box maybe some things changed with 6.0.4

[Center-Sun]

Hi @cybersecurity99 , It's worked in my suricata-6.0.4.
I noticed you added the kafka section at the end of suricata.yaml ,the things you need to know,this project is a eve output plugin,so you should edit the eve-log section in suricata.yaml like this:

  # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: kafka #regular|syslog|unix_dgram|unix_stream|redis|kafka
      filename: eve.json

      kafka:
        brokers: "kafka1:9092,kafka2:9092"
        topic: suricata
        client-id: suricata_client_01
        buffer-size: 1024

Hope it can helps you

[cybersecurity99]

Hi @cybersecurity99 , It's worked in my suricata-6.0.4. I noticed you added the kafka section at the end of suricata.yaml ,the things you need to know,this project is a eve output plugin,so you should edit the eve-log section in suricata.yaml like this:

  # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: kafka #regular|syslog|unix_dgram|unix_stream|redis|kafka
      filename: eve.json

      kafka:
        brokers: "kafka1:9092,kafka2:9092"
        topic: suricata
        client-id: suricata_client_01
        buffer-size: 1024

Hope it can helps you

Hi @Center-Sun sorry for late reply
But It worked , It was able to connect and create a topic but it is not sending any data .
It is not writing on disk or writing to eve.json which means it is sending data but I don't see anything in kafka .

14/4/2022 -- 17:56:45 - <Info> - --- Total Packets: 14841  Total Dropped: 0 ( 0.00%)
14/4/2022 -- 17:56:45 - <Info> - Alerts: 26
14/4/2022 -- 17:56:45 - <Notice> - Kafka produce finished: count=1227, dropped=0
14/4/2022 -- 17:56:45 - <Notice> - Producer finished: count=1227
14/4/2022 -- 17:56:47 - <Info> - cleaning up signature grouping structure... complete

I get this in suricata.log but my kafka is empty
When i check topic suricata stats I get this : suricata:0:0

Can you tell what may be the reason ? or any file I need to check to find issue.

This kafka issue tougher than finding One Piece :)

[Center-Sun]

Hi @cybersecurity99 ,i don't have any idea now, because according to the logs, it was working fine. Did you find any exceptions or errors?

[cybersecurity99]

Hi @cybersecurity99 ,i don't have any idea now, because according to the logs, it was working fine. Did you find any exceptions or errors?

@Center-Sun I tried to look for any errors but I don't find any . Also I am unable to locate where my logs going because it is not writing to eve.json on disk or to kafka . It just created topic suricata .

[cybersecurity99]

Also what's this client_id ?

[Center-Sun]

Also what's this client_id ?

it's a property of kafka client

[Center-Sun]

Hi @cybersecurity99 ,i don't have any idea now, because according to the logs, it was working fine. Did you find any exceptions or errors?

@Center-Sun I tried to look for any errors but I don't find any . Also I am unable to locate where my logs going because it is not writing to eve.json on disk or to kafka . It just created topic suricata .

This issue may be related to kafka broker🤔

[ImadYamane]

I am facing the same issue with suricata suricata-6.0.4. @cybersecurity99 have you found a fix?

[dependencies]
rdkafka = "0.25"
suricata = {version = "6.0.4-alpha.1", package = "suricata"}

[cybersecurity99]

I am facing the same issue with suricata suricata-6.0.4. @cybersecurity99 have you found a fix?

[dependencies]
rdkafka = "0.25"
suricata = {version = "6.0.4-alpha.1", package = "suricata"}

@ImadYamane
[dependencies]
rdkafka = "0.25"
suricata = {version = "6.0.4-alpha", package = "suricata"}

Worked for me

[cybersecurity99]

@Center-Sun Hi I wanted to know how we configure bucket size in this , I am not sure about the data rate it will transmit . can you help in this regard

[Center-Sun]

@cybersecurity99 Hi , can't configure now ,but this plugin based on librdkafka ,it's support many configuration property https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md . you can modify this plugin to support bucket size or others

[cybersecurity99]

HI @Center-Sun isn't files in src folder are hard coded
Like it's build on librdkafka
but it have only 4 field options
struct ProducerConfig { brokers: String, topic: String, client_id: String, buffer: usize, }

[Center-Sun]

Hi @cybersecurity99 , No other options are hard-coded , I only used these four options.