Skip to content

GSoC 2016 Proposal: WebSocket for syslog ng (Yilin Li)

Jump to bottom
LiYilin edited this page Mar 18, 2016 · 1 revision

Introduction

WebSocket for syslog-ng

syslog-ng is an famous log management infrastructure for users all over the world. Syslog-ng is rightly known as the "Swiss Army Knife of logging" because of its rich filtering capabilities, flexible configuration options. WebSocket protocol is a protocol enabling two-way communication between a client and server over a single TCP connection.
This project will add WebSocket as a destination for syslog-ng. The new destination can send log messages to an WebSocket Server directly. In addition, this project will enable the new destination to act as a WebSocket server, so the WebSocket client (such as a javascript client in a browser) could subscribe directly to it to get log messages. These will make sending log message privately or publicly much easier. This project will add WebSocket as a source for sysnlog-ng. So that it can receive log much more conveniently in some area such as IoT world.

Benefit to community

Having an abundance of destinations is one of the most important reason for Syslog-ng's popularity. Adding another destination based on WebSocket protocol will make it more welcome. And WebSocket indeed have some advantages over other protocols.

  • Unlike TCP, which is a streaming transport, WebSockets is a message based transport: messages are delimited on the wire and are re-assembled in-full before delivery to the application. This mechanism is more suitable for log message transportation.
  • After the initial handshake request/response, there is very little per message overhead. This makes it much better than HTTP protocol to act as a destination transport protocol for Syslog-ng.

With the two-way communication enabled, we can interact with the WebSocket destination and achieve some goals other destination couldn't achieve, such as allowing WebSocket clients to subscribe to the log messages by itself and to send message to the WebSocket destination to filter log messages. This is a feature that I discussed with mentor @Laci Mészáros and our earnest user @Fabien Wernli. You can found it in the mail list(mail 1 and mail 2). This is exactly what our users want.

I also plan to provide a WebSocket source. One of the used protocol in IoT world is websocket, so the devices could send log messages directly to a syslog-ng server. This will bring great convenience for IoT world. So syslog-ng will be more welcome in IoT world.

Goals for the project

The goals to accomplish during the GSoC project time-frame are listed below.

  • Developing a WebSocket destination.
    • The destination is configurable. We can configure the IP address, listening port and so on.
    • It supports client mode: In this mode, it acts as an WebSocket client and sends log messages to another Websocket Server
      • A sample WebSocket server will be provided to communicate with the WebSocket destination to make sure it is working properly in the client mode.
    • It supports server mode: In this mode, it acts as a WebSocket Server. It has a log messages buffer(The buffer size is configurable). The log messages sent to the destination are stored in the buffer (If the buffer is full then the oldest message will be overwritten). Other WebSocket clients(Such as a web browser) can directly connect to the Websocket Server to subscribe the message. The clients can send filter rules to the WebServer to filter the log messages it subscribe.
      • A simple web WebSocket subscriber will be provided to communicate with the WebSocket Destination in the server mode. This will demonstrate how to use the APIs to subscribe and filter the log messages. Other developers or users can implement applications with much better UE with this example.
    • The destination should support template.
    • It supports secure connection for transporting log message. This should be configurable.
  • A WebSocket source which can receive log messages directly from WebSocket clients.
    • The listening address and port of the source can be configured.
    • A simple WebSocket Client to send log messages to the WebSocket source should be provided to make sure the source is working properly.
    • secure connection for transporting log message. This should be configurable.
  • Ensure that the new features are robust and able to run steadily under all kinds of circumstances. Such as reconnecting automatically when it loses the connection.
  • The new WebSocket destination and source should be merged into the code base.
  • It should be well documented so that normal users can easily understand how to use it.

Tools, Libraries and Knowledge Needed

This section will list the tools, libraries and knowledge needed to implement this project.

  • I will develop a WebSocket server based on this simple websocket server written in python.
  • libwebsockets will be used in the the WebServer destination and source code to implement the WebServer client and server.
  • I will use javascript to implement a WebSocket client to subscribe and filter log messages.
  • I'll use syslog-ng module incubator as my staging ground.
  • I should be familiar with WebSocket protocol.
  • The C language to implement the WebServer Destination and GDB for debugging.
  • I should learn Flex and Bison to define how syslog-ng parse the options of the WebServer Destination.
  • Automake will used to make the new destination and source compiled correctly.
  • Git and github are used to manage the code.

I'm familiar with python, C, javascript and how to implement a new destination(because I take part in Improving "curl" module recently). For the success of the project, I should know more about the details of WebSocket protocol. and how to implement a new source.

Schedule

The Community Bonding Period: 23 April to 22 May

  • Getting familiar with syslog-ng's code base. Especially, I should understand the architecture of the destination and source modules.
  • Post my questions in the mailing list and IRC if I get some problems understanding the code.

The Coding Period: 23 May to 23 August

I'm planning to devote 30~40 hours each week. I’m certainly willing to make more efforts if the project is behind schedule.

  • Week 1: Design the configuration pattern that the destination and source will read. Post it to the community to get feed backs. Reiterate the design on the community feedback.
  • Week 2: Implement the WebSocket destination skeleton and a sample WebSocket server
  • Week 3: Implement the WebSocket destination client mode and enable secure connection and template support. Make sure that it can communicate with the sample WebSocket server correctly.
  • Week 4: Code review, documentation and testing of the above work, also a buffer for uncompleted work.
  • Week 5: Implement the WebSocket destination server mode.
  • Week 6: Implement a web browser WebSocket client to subscribe log message from the WebSocket destination in server mode. The new destination should be able to filter messages according the messages send form the client.
  • Week 7: Enable secure connection in server mode and template support. Make sure that the web browser WebSocket client can communicate with the sample WebSocket server in server mode correctly
  • Week 8: Code review, documentation and testing of the above work, also a buffer for uncompleted work.
  • Week 9: Implement the WebSocket source to receive log messages.
  • Week 10: Enable secure connection for the WebSocket source, write a simple javascript WebSocket client to prove it can receive log messages correctly.
  • Week 11: Code review, documentation and testing of the above work, also a buffer for uncompleted work.
  • Week 12: Merge the code with master branch. Make some pressure test, check memory leaks and so on to make sure the new destination is stable. Also a buffer period for all the coding period.
  • Final week: Students tidy code, write tests, improve documentation and submit their code sample. Students also submit their final mentor evaluation.

Deliverables

The final deliverables would consist of items listed below

  • A new WebSocket destination
    • It is stable and can run steadily. All kinds of exception, such as disconnection , can be handled properly.
    • It supports secure connection, server mode, client mode.
    • It is configurable and can meet most of the requirements of users to send log messages to a WebSocket Server.
  • A sample WebSocket server to communicate with the WebSocket destination to prove that it is working properly.
  • A sample web browser WebSocket client to show how to subscribe log message from the WebSocket destination.
  • A new WebSocket source.
    • It supports secure connection
    • It is stable and can run steadily. All kinds of exception, such as disconnection , can be handled properly.
    • It is configurable.
  • The code style should be good. Python code will follow most of the conventions defined in PEP8. C code will follow the GNU Coding Standards
  • README files that can be easily understood by normal users.

Why syslog-ng

I acted as an SA for many systems and syslog-ng really helped me a lot. GSoC is a good opportunity for me to know syslog-ng much deeper. This will help me to be a better user of syslog-ng. This will save me a lot of time during my SA career in the future. It is cool for me to contribute to my favorite project. The syslog-ng community is much more friendly than many other community. I can feel it from the elaborated GSoC guidance. So I expected I could learn a lot under your guidance.

About me

My name is Yilin Li, I am a postgraduate student in Graduate School of Chinese Academy of Science studying Computer Software and Theory. I've developed many systems with C and python during my campus life career. I love open source software. GSoC is a precious opportunity for me to contribute to a open source software that will be used by users all over the world. During entire GSoC time-line, I will take feedback and suggestions from my mentor and community. The code and entire process is public. With the help of intelligent developers and enthusiastic users , I'm confident that this project will be completed successfully!!!

Other commitments

During the project time-line, sometimes I will take some time to read some papers and maintain some servers. These tasks will take me about 15 hours a week averagely. I can handle them well and they won't affect the GSoC project. I have no other commitments in the GSoC period.

Clone this wiki locally