Skip to content

GSoC 2020 Proposal:Add support for the template() syntax in the kafka() destination

Jump to bottom
efg001 edited this page Mar 31, 2020 · 1 revision

GSoC 2020 Proposal:Add support for the template() syntax in the kafka() destination

About myself

During my current internship, I worked on implementing an Apache HTTP server authentication module that adds support for a new type of token developed by the security team.[C] and a Linux PAM module to allow SSH authentication base on an open-source project.

I have experience in C and parallel programming/mutexes

Time Zone: CDT

I don't have any other commitment for the summer besides a potential short conference trip, but will have other commitment after mid-august and plan to finish the project by then

Project overview

When sending a message to Kafka, in the Java implementation updated early 2019, topics that was gotten through getTopic() method can be template, as inside the option class, there is a templateOptions hashmap that allows user to define a template. However, in the C implementation update by the end of 2019, the topic seems to be directly copied using g_strdup and thus will not support template syntax. related code:

https://github.com/syslog-ng/syslog-ng/blob/7d5fa862b0856d52ba350589cb013cf90a40b9bb/modules/java-modules/kafka/src/main/java/org/syslog_ng/KafkaDestination.java

https://github.com/syslog-ng/syslog-ng/blob/a5b1a540a485744296b5a79b7e1d806377371bc1/modules/java-modules/kafka/src/main/java/org/syslog_ng/KafkaDestinationOptions.java

https://github.com/syslog-ng/syslog-ng/blob/18b69c763bac1cac33067ad1385f6c51cc39303e/modules/kafka/kafka-dest-driver.c

https://github.com/syslog-ng/syslog-ng/blob/09a10460cd7a8d5fd91a1edbc3d57060d6116297/modules/kafka/kafka-dest-worker.c

Why the project is interesting for me?

I will need to work on Kafka related projects this Fall semester and have spent some time studying Kafka. I am very interested in working with experienced engineers from syslog-ng to learn and work with Kafka as well as contribute to syslog-ng. I received confirmation about working on a Kafka project in the fall last week and started late on the proposal but I am very committed and will do whatever I can to finish the project in the summer.

What are the knowledge areas required for the success of the project?

  • Familiarity with syslog-ng
  • Familiarity with kafka
  • Familiarity with C
  • Familiarity with parallel programming/mutexes
  • Java

I am comfortable with

  • Familiarity with C
  • Familiarity with parallel programming/mutexes
  • Java

By the summer, I need to study

  • syslog-ng
  • kafka, librdkafka

Project timeline

By summer:

  • syslog-ng, kafka,librdkafka

  • Define the actual scope of the component and a complete timeline(2 days to a week)

  • Setup working example of syslog-ng-kafka in Java and C(1 day to 3 days)

  • Study the Java implementation that supports template syntax ( 2 days to a week)

  • Fork, add support to the C base implementation base on how it was done in Java (a week)

  • Manually test the fork and debug(a week)

  • Brief code review, feedback, debug(1 day to 3 days)

  • Write test and debug(2 days to a week)

  • Documentation(1 day to 3 days)

  • Code review, evaluation, pull request(a week)

Clone this wiki locally