-
Notifications
You must be signed in to change notification settings - Fork 0
GSoC2015 Proposal: WebSocket for syslog ng (tsboj17)
Syslog-ng is one of the most famous logging daemons. It can receive log messages from a lot of sources (file, tcp/udp, zmq) and can send logs to lots of destinations (amqp, sql, tcp). It works on lots of platforms. The WebSocket is a wordwide full-duplex tcp-based protocol and it has been designed to be implemented in web browsers and web servers, but it can be used by any client or server application. It would be great if Syslog-ng could send and receive log messages through WebSocket.
With Syslog-ng including WebSocket, it would have one more source and destination pair and be more attractive which would cause more people and organizations to use it. Nowadays the WebSocket is very worldwide, for example a lot of browser game is based on it so it has a big future and if Syslog-ng could receive logs from this protocol that would open up more opportunities. Also the API is being standardized in Web IDL by W3C.
No other solution exists that provides true bidirectional communication like WebSockets, there is a persistent connection between the client and the server and both parties can start sending data at any time. It's built on TCP, message based communication which uses HTTP in application layer. Because of this, it's very simple to use and understand.
The communications go through port 80 which is the port of HTTP. This is one of the best benefit because this port is common. That means the sysadmins don't have to open any port on firewall because the port 80 is usually opened.
- The logging is very important in the sysadmins' work. But what if the user wants to know about some kind of log messages like wrong authentication? With WebSocket supporting it it would send these kind of messages to the custom HTML page. Here the logs can be checked
- The Syslog-ng can receive logs from applications which are using WebSocket
- It can be a mobile monitoring solution too. Syslog-ng sends logs through WebSocket and the mobile app receives through it
Summarized: The IoT is more and more widely known which means every device we use goes to the internet and they can be configured from that. How can we check these devices? They can communicate on a lot of protocols (http, xmpp, mqtt etc.). One of them is WebSocket. Because of this, the proposal can be useful for checking different devices after this project because the Syslog-ng can receive and send log messages on it.
If this project is successful, the secure WebSocket could be implemented with minimal effort, also it uses TLS for secure connections.
I use Syslog-ng almost every day, but right now I don't know how it works, I see it as a black box. After the project I minimally could know how the code works. Needless to say, I would:
- enrich my development skills deeper (coding and tests in C mostly)
- know more about HTTP protocol
- learn new technologies (WebSocket)
These experiences are useful for the future because I want to be a DevOps engineer.
I want to implement the WebSocket as simple source and destination like tcp/udp, file etc.
For the sake of the cause I would like to use the libwebsockets library which is a lightweight pure C library built to use minimal CPU and memory resources, and provide fast throughput in both directions. I think the developers have to write tests, especially unit tests. With those, the functions can be tested and the problems are avoidable. The functional tests are requisites too because the work can be proved.
As I noticed the receiver does not have any global solutions to handle the blocked loops. I'm planning to create a few functions where I create loop with exit signal:
ws_consumer_loop()
ws_request_to_stop() // obviously there is no loop in this functionThe consumer_loop handles the receiving.
I use the main_loop_create_worker_thread() with the proper parameter (previous functions for example) and this could handle the proper interrupts.
Without being exhaustive I would write these functions:
ws_dd_init()
ws_dd_new()
ws_dd_connect()
ws_dd_set_host()
ws_dd_set_port()
ws_dd_set_http_header()
ws_dd_add_header()
ws_worker_insert() // --> This is the sending function. Here i want to use libwebsocket_callback_on_writable() in libwebsockets.
ws_dd_disconnect()
ws_dd_deinit()
wss_dd_free()At the header, it is a good idea to use templates because these data can be generated by dynamically.
I would make an own struct whose super class is the LogThrDestDriver. Something like this:
typedef struct
{
LogThrDestDriver super;
gchar *host;
gint port;
//of course it is not the full struct
} WebSocketDestDriver;As I noticed from the syslog-ng code, I could register my own function as you can see at the example:
self->super.worker.thread_init = ws_worker_thread_init;
self->super.worker.thread_deinit = ws_worker_thread_deinit;
self->super.worker.insert = ws_worker_insert;The ws_worker_insert would handle the log sending.
I am not sure that the bi-direction is needed in my case since I don't have any use-cases for that yet. I think this is a nice-to-have function at the logging.
- Apr 28th – May 23th
- Getting deeper knowledge of WebSocket and Syslog-ng source and destination
- Getting familiar with Syslog-ng code base
- May 23th – Jun 23th
- Designing the structure of new source
- Implementing WebSocket source to Syslog-ng
- Writing unit tests and end-to-end tests
- Jun 24th – Jul 26th
- Designing the structure of new destination
- Implementing WebSocket destination to Syslog-ng
- Writing unit tests and end-to-end tests
- Jul 26th – Aug 5th
- Measuring the code: code coverage, leak hunting, etc.
- Aug 6th – Aug 17th
- Finalizing the code
- Review and code cleanup → The code has to fit to the rules of clean code (of course not so strictly)
- Performance measuring
I don't expect any longer period away from keyboard. Now I do not have any vacation or traveling plan. I am working part-time so I have to work 3 days (24 hours) in a week but the rest of the time I can work on my project.
On my GitHub page you can find the tasks and the solutions.
I am a Computer Engineering student at Budapest University of Technology and Economics. I expect to graduate in 2016. I have been using Linux since 2010. I am a system administrator in Schönherz dormitory. I mostly use Ubuntu (and forks of it) on my desktop machine, but on servers I use Debian. I have experience programming in C, C++, Python. Besides of these, I used Java, PHP, C# but these were only for homeworks. My favorites are the former ones. The Test Driven Development is not far away from me either. I have developed a bigger python project in TDD for 4 months. It was meant to be an open-source project but it failed so the code didn't get to GitHub.
Because of my system administrator title, I am familiar with Syslog-ng configuration. I know the syntax, the usage and how to debug. I like to learn new things, should it be protocol, programming language, code base, does not matter. I am an enthusiastic guy who always does his job conscientiously.
Last but not least I want to express my gratitude to László Mészáros and Ádám Mózes for sharing their helpful thoughts about this project with me.