GSoC 2015 Proposal: syslog ng as a command line tool (kvch)
Syslog-ng is a system logging application available for a wide range of operating systems. It is capable of sending, filtering, parsing, rewriting and forwarding logs to various destinations. It runs as a daemon process on systems and configured via files.
The projects goal is to extend its functionality by making it able to run offline as a command line tool reading incoming logs from stdin, transforming the input the usual way and forwarding it to stdout.
- new drivers: stdin source, stdout destination
- configuring parsers, rewrite rules, message manipulation via command line options
- not breaking current functionality
In order to receive logs through stdin and forward it to stdout destination, two new drivers are needed: an stdin source driver and an stdout destination driver. Right now it is possible to configure a basic source and destination using the file driver. The main problem is that syslog-ng does not stop after it has read all of the input. Reading from stdin should stop when it processes all of the incoming logs or the process is terminated e.g. in case SIGINT is sent. This can be solved by sending ACKs to the source, if a log message was processed.
Every syslog-ng instance has a persist file, by default its name is syslog-ng.persist. Upon restart the last read log is recorded in this file. In the given persist file the last read line by input - in case of files or databases - should be recorded. This way syslog-ng is prevented from rereading lines. Writing to the persist file would be necessary rarely when running as a CLI.
If a persist file is needed and a syslog-ng process is running on the system, each instance needs separate persist files. Otherwise the command line interface would use the process's file and a process would not be able to write that file until it is restarted
Configuration is written into files in the usual syslog-ng configuration language. This language has to be extended with new keywords, in order to add these new drivers. For example stdin() could be the stdin source and stdout() could be the stdout() destination.
A few restrictions have to be introduced in the configuration. If there is an stdin() in the source definition, other sources like internal() are not allowed. This makes the source definition clearer and more understandable. The same applies for stdout(). Only stdin and stdout are allowed in the same log paths. If the configuration file contains other sources, destinations and log paths apart from stdin and stdout, those ones are ignored.
@version: 3.6
source s_stdin { stdin(); };
destination d_stdout { stdout(); };
log { source(s_stdin); destination(d_stdout); };
To make the tool more user friendly it would be possible to configure with command line options. This means that new command line options should be introduced. For example it would be easy to add a filter to the logpath using the --filter option.
$ syslog-ng --filter=host:"host1"
$ syslog-ng --filter=host:"host1ORhost:"host2"
Most of the currently available options for syslog-ng can be used, too.
An interesting question rises in connection with usabilty. As I mentioned above, the sample configuration would be present always if syslog-ng is started as a command line tool. Time could be saved if it would be possible to tell syslog-ng that it should run as a CLI without editing the config file.
A possible solution is to add an option e.g --cli which would include this simple config in the given syslog-ng configuration file. In this case every other source would be ignored and filter and parsers could be configured via command line options.
Another method is to check the given command line options and if it contained command line specific instructions syslog would load the cli's configuration.
In verbose mode the program writes its logs to syslog-cli.log to help debugging and serve additional information about each run, for instance log statistics. This way, log messages are separated from syslog-ng's messages.
While coding I would like to write unit tests, in order to make sure that functions work properly. Besides, for the sake of maintainability, further testing is needed.
- Make sure that the given command line options generate the proper configuration
- Make sure that every log message is processed as the configuration suggests
- Pass regression tests
The main benefit that the current syslog-ng functionality would be available from command line. In some cases there is no log to be collected from the network, but from the standard output of programs. This tool makes it possible to deal with logs in such scenarios.
The CLI could be a valuable tool in transforming log messages. It would be possible to rewrite, parse messages without sending those to syslog-ng and forwarding to e.g. file destination. The syslog-ng daemon could be replaced by the command line tool in such use cases.
Furthermore, more complex rewrite rules and filtering could be tested before adding those to syslog-ng process configuration. The current options to test the configuration before starting the process is not sufficient, because it is not possible to test the functionality without starting the process. Reloading the configuration time after time can be a pain in the neck. This tool would make syslog-ng more user-friendly.
- no need to run as a daemon process
- transforming log messages via command line
- testing filters and parsing becomes easier, more user-friendly
In June I am going to take 2 exams at university. But besides that I am free during summer.
- Community bonding time:
- reading necessary documentation
- getting to know the code base
- getting to know the mentor
- planning the smaller details of the tool
- Week 1-3: implementing the drivers
- Week 4-6: implementing command line options
- Week 7: revisit problematic lines, re-factoring the code
- Week 8: integrating
- Week 9: writing end-to-end tests
- Week 10: writing documentation
- Week 11-12: buffer period
My name is Noémi Ványi. I am 21 years old. Currently I am studying Computer Engineering at Budapest University of Technology and Economics, Hungary.
I am passionate about programming and I like coming up with solutions for new problems. I am striving to write clean code. I am keen on UNIX and UNIX-like systems. I am familiar with C, C++, Python and bash.
My main motivation is to deepen my knowledge of syslog-ng. Right now it runs on my laptop and my home server with basic configuration. I find it a handy tool with great capabilities and I would like to make the most out of it.
I like using the command line, and I would like to add new tools to the command line world. My current hobby project is a command line tool, too. But I would like to contribute to a more serious open source software which creates value and used by many people around the world. Also, it is high time I give something back to the community.
Last but not least, I would like to do it for fun. :)