Skip to content

GSoC2016:syslog ng as command line tool(Amey Jain)

Jump to bottom
Amey Jain edited this page Mar 26, 2016 · 1 revision

Basic idea:

Syslog-ng currently reads log messages from the sources as directed by the source block in the configuration and after processing these messages are written back to the destinations directed by the destination blocks. The idea I want to work for is to make syslog-ng as a command line tool in order to process the messages taken from STDIN and its output displayed to STDOUT. Like in existing syslog-ng here in command line we do not need runtime monitoring of the files i.e,. we do not need to check the source again and again for changes. A new command line tool will be developed which will take the log messages from stdin and print the processed output to the stdout. The new tool will use the existing functions of the syslog-ng in it and it will connect those functions with command line. Thus we will be able to harness the capabilities of syslog-ng with runtime input through command line also.

Why I am working for it?

  • I will learn and experience software implementation process. How an idea in community and turns into software.
  • I am greatly fascinated by how open source community works on a software. I use valgrind,firefox and many other tools like it. I use Ubuntu 14.04 and have seen the strong community support of it. The community of syslog-ng is also very helpful.

Benefits of command line tool:

As a command line tool we will be able to use the features of syslog-ng. Apart from message transfer between remote locations syslog-ng as command line tool can give us features like CSV parser and patterndb. We can use these functions to simplify piece of texts in STDIN.

Example case: Recently in a college fest I was given a text file of whatsapp chat and I was asked to write all the mobile phone numbers from those chats. There were about 52 pages of chats. I did not want to just open that file and note down the numbers. So I asked one of my college senior and we made a regular expression program that scanned the file and print all the numbers. In cases like this we can simply input the text into the command line syslog-ng with regular expressions and parse the output that we need. The basic thing that command line syslog-ng will be bring to us is that it will able to support the features like templates, macros, parser and patterndb together.

The command line will have the following features after it has been completed:

  1. A new compiled binary will be there which will handle all the arguments given from the command line. It will take all arguments and the messages of input. The command line will be having: -file flag:to specify the location of the files for templates, parsers, etc -oper flag: to enter the operation that the user needs to enter. In this case the file flag will be overridden. These two flags will also enable the user to either enter the operations(templates,parser etc) on the runtime or take them from a specified file.

  2. User will be able to process the messages with parsers, templates and patterns that user specified in the command line or he can write them into a file. But still use of command line will be there. Since the set of parsers, templates and patterns all together will be long to enter as a string every time, user can write them once and store them in a file with unique reference and then use it whenever he/she wants with that reference.

  3. For input user will have an exit command which he can enter each time the input is completed.

  4. An internal() function will also be there that records the messages of syslog-ng command line and store them in a buffer for current session. After the input is complete these internal() messages are displayed in stdout only along with the other outputs. It will help the user to see what are the errors in input.

Implementation details:

  1. Two new files handler.c and handler.h which will accept the arguments of the operations from the user, check those for syntax errors and then perform those on the given input.This script will also be able to display and edit those options from the header.h file.

  2. Note: The handler.c will take the input formats as same given in syslog-ng.config file.

  3. This handler.c file will also include lib/cfg-lexer , lib/cfg.h and lib/cfg-parser files which will be able to check the syntax of the options given to handler.c file in command line.Also In our case the CfgLexerKeyword main_keywords[] in cfg-parser.c file will not contain unnecessary keywords like source and destination. Also the errors from here will be handled straight to the internal(). internal() will be responsible to print these error to stderr.

  4. Also the handler.c file will include a function that will specify the operation(parser,templates etc) to be taken from the command line on runtime or to read them from a by default user file. User will also be able to write its operations on that file and use it again in future.

  5. After correct options are handled to the handler.h file handler.c will be handling these options to the main.c file. main.c file will be responsible for the main action. This main.c will be responsible for reading the templates and applying them to the stdin inputs.

  6. I will be adding new feature and it will be able to handle: CSV parser, filter, Templates, Matching expressions with the help of regx, If got time then patterndb also.

  7. An internal() will be there. It will record all the messages generated by the syslog-ng command line and will be handling them. This internal function will be responsible for generating the error messages in the in between the processing and displaying it to stdout.

Timeline of the project:

April 22 - May 22:

  • Getting familiar with the community. Gain more knowledge on the scope and utility on my project.
  • Get to know more about the syslog-ng itself.
  • Gaining a good hold on syslog-ng source files. And get a rough sketch map in mind of all the functions and the variables that I may use in future.
  • Get familiar with the datatypes used in the project.
  • I will be having my examination during this period so I can not do many things this time.

May 23 - May 29:

  • Start coding out the handler.c and handler.h files. They will be containing only the command line interface with them. I will be making the command line interface which will be able to accept the operations and also the inputs.

May 30 - June 9:

  • I will not be able to work this time. I need to attend my NCC(National Cadet Corps) camp.

June 10 - June 28:

  • Including the lib/cfg-lexer, lib/cfg.h and lib/cfg-parser in the handler.c and making the command line able to check the options given with the help of included files.
  • Also I will be making a new function for user to choose either to input the operations directly to the command line or to take them from an input file.

June 28 - August 5:

  • Making the main.c file. This file will be including main operations to be performed on the input.
  • Using the specific files for templates, parsers in the this file I will be performing the operations.
  • I will be in constant contact with my mentors, because of lack of the knowledge of code base. August 6 - August 16:
  • Testing the command line tool.

Required field of knowledge:

  • For handler.c and handler.h: Familiarity with C language which I am.Working with command line arguments and parsing those arguments to make files which will be creating operation configuration.
  • For error checking: Using the included header files. Linking the configuration file or the operation input given to the command line and checking those operation input with included header file and functions.
  • For main.c file I need to understand how a particular operation like parsing is applied to a piece of input. How the input is passed and how the output returns from the function.
Clone this wiki locally