GSoC2015 Proposal: syslog ng as a command line tool (Asad)

Syed Asadullah Hussain Master in Computer & Systems Engineering from TU Ilmenau, Germany

Prior to my Master, I have done bachelor in Electronics Engineering (Computer Networks) from National University of Sciences & Technology, Pakistan. I have worked for 2 years on projects by Silicon Valley companies such as FireEye, Cavium, Dell and Intel from the platform of Ebryx & xFlow Research. Where I have acquire skills ranging from Software Development (C, Python), Quality Assurance to Network and System Administration.

Proposal:

My motivation is to contribute to opensource development and as my main motive is to pursue a career in systems engineering. Syslog utility development is very exciting for me. This project will involve going through the main code path of existing code base, which I am looking forward to. As I love using printf and oprofile to trace code :). I have already downloaded the tarball and am going through the code coverage.

Problem Statement:

We will develop a new source (source driver) where the input from command line will be stored, this source will be connected to a destination through a "log path" which should be based on parameters of our command line tool.

syslog-ng daemon "actively" collects log messages from a variety of "sources" (can be files or remote hosts or stdout of a program) and is capable of routing these messages to a number of "destinations" (can be files or remote hosts) for further processing but under normal conditions these sources/destinations are defined in the syslog config file before starting the daemon and the daemon has to be restarted every time we make a change in the config file.

The task of this project is to provide an alternative to the "active" log collection by the daemon. Without starting the daemon, the user should be able to provide any input (file) which can be like the input from any of these sources to our command line tool and it should be able to able to process it. Destination: Now the command line tool should also support parameters through which users will be able to specify the destination to which they want to send the file.

Plan:

As a reference starting point the patch, is very useful.

0. Implement a dummy Source and Destination & propose a solution to the problem of when to stop syslog-ng? Timeline: April 1-20

Progress:

Implmented a simple stdin source, which reads a log file line-by-line and outputs it on stdout. https://github.com/asadpiz/syslog-ngcmdtool.git

1.STDIN: Understand the entire flow of code. Which means that one should be able to execute all code paths (documentation + test suites). The purpose of this task is to identify the point in code where the client sends the message to the server where it is sorted and stored. This will be function where "stdin" data will be stored into syslog i.e., our input will be read as one message at a time and will be stored using the same function calls the server uses to store client messages.

Deliverable: Our command line tool has to read the input (file or remote message) 1-by-1, just like syslog-daemon does. We just have to parse these messages one by one to the "function" (in the source code). This is the function where the daemon sends it's messages for further processing.

Timeline: 1 May to 20 May

2. STDOUT: This task involves identifying the point in code where the syslog daemon enables verbose logging for output logs to command line. Verbose logging displays the log messages as they arrive. So the same commands can be used to sort and display messages on command line.

21 May to 31 May

3. Input/Output: This is the main part of the task here we have to develop a command line utility "syslog_cmd" which will take input and output from command line and direct to to appropriate function calls idenified in steps 1 & 2.

Solve the problem of when to stop syslog-ng?

1 July to 30 July

The utility will also be able to read syslog-conf file from command line.

Relevant Experience: (I cannot post the complete projects due to legal bindings but I have uploaded an openvswitch fork with my contributions, I have mentioned exact files against each task):

https://github.com/asadpiz/Custom-Netdev-Provider/

I have extensive experience with C programming, especially relevant is the experience of modifying the command line utilities of ovs i.e., ovs-vsctl, ovs-ofctl.

OVS development:

I have modified the command line ovs-vsctl/ovs-ofctl/ovs-dpctl to incorporate custom fields. (asadpiz/Custom-Netdev-Provider/blob/master/lib/ofp-util.h).
    The “ovs-ofctl add-flow” command stores the rule into the “ofputil_flow_mod” structure with values such as source/destination IPs, ports and the openflow action. This structure is passed to the add_flow function.
    In add-flow function the “action” of the rule is extracted and verified by matching it against the supported openflow actions e.g., normal, drop, flood and output. The function proceeds only if the action is supported otherwise error is thrown
I have also modified the kernel path of OVS (for datapath stats collection). Can't upload due to legal reasons but here is the code flow:
I have ported OVS on both Cavium & Intel platforms. (Custom-Netdev-Provider/blob/master/lib/netdev-intel.c & netdev-intel.h)