Skip to content

GSoC 2018 Proposal: HTTP batch sending (NikitaDoroshkin)

Jump to bottom
NikitaDoroshkin edited this page Mar 27, 2018 · 1 revision

About project

HTTP batch sending project is HTTP destination module improvement which enables batch requests for HTTP destinations that support such. This project allows to group separate HTTP calls together into a single HTTP request, instead of sending each call separately. The result of this project is an improved HTTP destination solution (libcurl based) that will make possible to send batched HTTP requests.

Why it is useful for syslog-ng

Syslog-ng is used to collect logs, process them and deliver to different destinations including HTTP. The current problem is that syslog-ng now uses separate HTTP calls for sending single message, which results in a certain amount of overhead. It’s a common situation when there is a set of relevant data - in case of Splunk it can be a set of events - and, therefore, it is naturally to pass this data in single bunch than to do multiple requests (in case of Splunk batched events are sent to HTTP Event Collector on Splunk Enterprise or Splunk Cloud).

All in all, the main advantages this project will bring to syslog-ng are as follows: * HTTP batch sending decreases overheads by single request than by multiple. * It provides better semantic basis of this approach in case of sending relevant set of data. * Also it improves flexibility of sending HTTP requests in system, since you can choose most suitable approach.

Why it is interesting for me

I’m seriously passionate about applied computer science and this project is a great opportunity for me to apply my knowledge and skills, especially those that I’ve received during programming and computer network courses at the university and self-study. I think that it’s a great opportunity to apply theoretical knowledge and practical skills in real world full-scale project. Study is good in case that it gives you theoretical background and the opportunity to develop small educational projects, but I want to participate in something really significant and get the invaluable experience of working with a large, open-source codebase.

Except programming, one of my strenghts is computer networking including knowledge in network protocols, so I decided that this project best suits me. I believe that to participate in the development of such a large open-source project with the guidance of mentor is the best way to get experience and become a part of open-source community.

Project goal

The goal of HTTP batch sending project is to create reliable, effective, well-tested and fully documented functionality, that will make it possible for syslog-ng HTTP destination solution to make batch HTTP requests.

Skills for successful implementation of the project

Knowledge areas required for the success of the project are: * libcurl - Familiar * C - Familiar * syslog-ng (destination module, threading) - Beginner * Community - Beginner

Project Timeline

April 23 - May 14

  • Familiarize with community and mentor.

  • Understanding of the syslog-ng project related documentation and codebase.

  • Increase familiarity with current HTTP module.

  • Implementing a design of new module

May 15 - May 23

  • Increasing knowledge in syslog-ng module implementing

  • Designing and prototyping functionality.

  • Discussing results with mentor.

May 23 - June 10

  • Core functionality implementation - combine messages in batch and send in a single HTTP request.

June 11 - June 15

  • Fist evaluation.

  • Discussion of future features.

  • Planning work.

June 15 - July 8

  • Continue work on project:

  • Finish completing the functionality - add flexible configurations.

  • Integration with syslog-ng.

  • Testing, some improvments.

July 9 - July 13

  • Second evaluation.

  • Getting feedback and discussion of work.

  • Planning work.

July 13 - August 1

  • Work on project:

  • Finish integration into syslog-ng.

  • Integration and testing with Splunk’s HTTP API.

  • Minor improvments and tests.

  • Self evaluation.

August 1 - August 6

  • Minor fixes, tests, improvments.

  • Documentation, examples, code review, user guide.

  • Evaluate with mentor.

August 6 - August 14

Final submission and evaluation.

August 14 - August 21

  • Mentors Submit Final Evaluations.

Results

  • HTTP destination module with ability to send multiple messages in single batch.

  • Integration with HTTP destinations with batch requests support.

  • Good tested solution.

  • Documentation and user guide with relevant examples.

About me

  • Name: Nikita Doroshkin

  • Address: Belarus, Minsk, 10 Oktyabrskaya street, Apt. 907

  • University: Belarusian State University

  • Faculty: Faculty of Applied Mathematics and Computer Science

  • Speciality: Applied Computer Science

  • Education level: Second year undergraduate

  • Languages: C/C++, Java, C#, Python, JS, SQL

  • Contacts:

Clone this wiki locally