-
Notifications
You must be signed in to change notification settings - Fork 0
GSoC2014 Proposal: ZMQ transport (arunmathew88)
Name : Arun Mathew
University : Indian Institute of Technology (IIT) Bombay
Phone Number : +919167518620
Address : Room #C304, Hostel 13, IIT Bombay, Powai, Mumbai, Maharashtra, India. PIN-400076
Timezone : UTC/GMT + 5:30 (India Standard Time)
IM & Contact : IRC nick - arunmathew88
Skype - arunmathew88
e-mail - arunmathew88@gmail.com, arunmathew@cse.iitb.ac.in
URL : http://www.cse.iitb.ac.in/~arunmathew/
Languages : C/C++[advanced], Java [intermediate], Python [beginner], PHP [intermediate], JavaScript [intermediate], bash [intermediate]
I am pursuing Masters Degree in Computer Science and Engineering in India's premier Institute. Previously I was a Senior Software Engineer with Yahoo! in their Bangalore office. I also share the responsibility of Administering the Department IT Infrastructure. My research interest is in the Distributed Event Based Systems [DEBS] domain, which follows from my work experience on high volume, low latency distributed event stream transports. An open source enthusiast, who has used a lot of open source projects and made small contributions to the open source projects. I started programming from by 5th grade and has won programming competitions at state level in my 9th and 10th grade in C/C++. Over an year of my work experience was on Yahoo!'s high volume data pipeline - Data Highway, which was written entirely in C++. Specifically I was working on Data Highway's streaming pipeline called Offramp.
syslog-ng already has many destination drivers and source drivers. AMQP has only a destination driver and not a source driver. This project aims at implementing a transport based on Zero MQ, a high performance low latency messaging library for intra/inter process and across the system messaging. This will enhance the performance of client - server and client - relay - server configurations of syslog-ng, especially when there are multiple servers to which we need to send the logs to. [As in the case of high availability].
- A transport implementation for zmq() to facilitate basic read/write methods.
- A destination driver zmq() to send messages using zeromq transport.
- A source driver zmq() to receive messages via zeromq transport.
- Appropriate configuration options & flags to operate the zeromq connection in
- point to point [client - server]
- pub-sub mode [client - multiple destination]
| Apr 21 - May 19 |
Understand the syslog-ng source/destination driver codes & design in detail. Learn any coding conventions used. Understand the polling mechanisms and other designs used in syslog-ng by looking at modules/affile modules/afsocket etc and their dependencies. |
| May 19 - May 27 |
Implement the zmq transport methods in lib/logtransport.h Specifically implement following functions in lib/logtransport.h |
| May 28 - Jun 3 | Design polling and other protocol layers as necessary to facilitate the source and destination drivers. |
| Jun 4 - Jun 14 | Implement polling and other protocol layers. |
| Jun 15 - Jun 22 | Initial Implementation of zmq() destination driver. Tested against a test zmq server, with test cases to verify the basic working on a point to point configuration. |
| Jun 23 - Jun 27 | Mid term evaluation. |
| Jun 28 - Jul 13 | Initial Implementation of zmq() source driver. Tested against a test zmq client, with test cases to verify the basic working on a point to point configuration. [More time here than destination driver since this will need to make use of polling and other structures, so chances of bugfixing] |
| Jul 14 - Jul 17 | Integration testing with basic configuration. Actual syslog-ng agent with zmq source and destination drivers. |
| Jul 18 - Jul 29 |
Adding configuration options and flags to the basic source and destination driver implementations so that we can tweak the zmq parameters. Adding the additional test cases to test each parameter. Add configuration options for pub-sub model to support multi destination sending using transport. Add test cases for the scenario. |
| Jul 30 - Aug 10 | End to end testing of all scenarios. Bux Fixing. Test case enhancements. Documentation and buffer time. |
| Aug 11 - Aug 18 | Pencil Down. Cleanups, documentation enhancement, etc |
I do my masters in Distributed Event Based Systems domain, of which publish subscribe systems are a big part. I have worked with padres which is a very common pub-sub middleware, and have a brief work experience with zeromq which we tried in our submission to DEBS 2014 conference Grand Challenge. We used sockets for our Grand Challenge submission whose code is hosted in https://bitbucket.org/debs/c-smartplug/overview. Previously I was working in Yahoo! 's Distributed event stream pipeline called Data Highway - Offramp [written in C++], which streamed apache event log lines as events from front end web servers across the globe to analytics teams. In Offramp we used Yahoo! proprietary proxyio messaging system, in this project we need to use zmq. I have been using Linux for the past 8 years and uses the logs heavily due to my role as a Departmental sysad administering 200 Linux nodes and 8 servers.
This proposal if accepted would be a great opportunity for me to contribute something significant to open source. I believe with the past experience I have and familiarity with the components, I should be able to do well.
My current semester ends by April 19th and the next semester starts in July end. Between this I might go to my home town for a week, but I would be carrying my laptop and being at home wouldn't affect work progress. In the next semester I would be having my Thesis work only, since I would be done with all my course requirements. Hence there would be ample time to spend on the completion of this project.