-
Notifications
You must be signed in to change notification settings - Fork 466
GSoC 2016 Proposal: Web based syslog ng configuration editor (Steams)
Syslog-ng is an enhanced and highly configurable logging daemon which supports a wide range of input and output methods and data formats. It has the ability to monitor sources for messages which it proceeds to process, filter and classify as the user has configured, before routing them to a final destination, either locally or on another server entirely.
Syslog-ng interfaces with the user entirely through its configuration file. Here users may, define multiple sources from which they are interested in collecting log messages, define filters and rewrite rules they wish to run the log messages through, define destinations they wish to route the messages to, and tie these all together into log paths, which specify an origin, path,processing and destination for log messages. It also allows the user to invoke a vast array of options that modify the software's behavior.
Syslog-ng's configuration language would appear to be created by programmers for programmers, as its syntax closely mirrors that of the C programming language, complete with curly brace block scopes and semi colon delimited statements. This may cause writing the configuration file to be an unfamiliar and uncomfortable endeavor for users inexperienced with programming, and possible even for those who are programmers, but wish to seek respite from having to write C whenever possible.
Additional complexity is introduced in the form of vast array or options, drivers, parsers and other objects and functions at the users disposal. Each of these may then accept further functions and options. These functions provide invaluable facilities to the user which they would surely desire to integrate into their configuration, given that they are willing spend the time to research and utilize these features, all of which are present in it's 360+ page administrator guide.
As such, this proposal outlines plans to create not only a drag and drop interface for creating configs that abide by the syntactical rules of the configuration scheme, but one which also provides appropriate prompts and tool-tips to ensure users don't miss out on nor misuse the useful features of the software, tucked away in the corners of the admin guide.
This project will allow users to quickly setup new and complex configurations for syslog-ng. It will provide them with a way to visualize the paths and processing through which their messages flow, in a similar vein to network diagrams. This new interface thus lowers the barrier to entry for new users to the system and may contribute to an increased in it's number of users. The visualizations this project will enable would also create a new medium through which system administrators can refer to and share the different possible structures of their syslog-ng configurations.
The success of this project depends on being able to determine and put into place the facilities and functionality to provide the end user benefits stated above. These facilities and functionality are listed below.
-
A database of all the objects and functions that can be utilized in the syslog-ng config file, their relationships to each other and how they are actually represented as code snippets.
-
A font-end web interface for making additions and modifications to this database.
-
A front-end ,drag and drop based, web interface for creating, editing, grouping and relating configuration objects graphically.
-
A set of front end facilities to convert these visualizations into actual configuration snippets and vice versa.
-
A set of front-end ui functionalities to support user discovery and appropriate usage of the objects and functions available to them, through intelligent tool tips, prompts and meta data based searching.
Completing this project will require a strong grasp of or skill with the following technologies and concepts.
- Database Design
- Back-end web development
- Ui design
- front-end web development
- the syslog-ng configuration language
- Bison and Flex, grammars and parsing
The following outlines components within the structure of my intended solution to the problem at hand.
A database of all the objects, functions, and their relationships that can be used in the config file.
In order to enable smart, context aware assistance to users on the front-end, we need to keep models of each object and function, their associated meta data (name,description etc), and relationships to other objects (required params, optional params, options etc).
The different kinds of objects present in syslog-ng configuration specification are:
- Source drivers
- Destination drivers
- Global Options
- Macros
- Parsers
- Functions/parameters
and then there are object types which users can define by utilizing the above objects:
- sources
- destinations
- Filters
- Rewrite rules
This database will also need to store a representation of the code required to create these objects in the configuration language, in a format that can be parsed by the front-end and replaced with user data. For example, sources would have stored a specification akin to "source |sourcename| { |drivername|( |parameters| |options| );};", which would be instantiated to "source s_local {file("/dev/log", follow-freq(1) );};".
I intend to implement the database schema in a postgreSQL database.
A rest Api to interface with the database allowing to to be easily extended. I am capable of implementing this api in either the java play!, nodejs express, or ruby on rails frameworks. But intend to work in ruby on rails if there are no objections.
Front end factories for each object type will be written to consume the information in the database through the rest api and use this information to generate constructor functions which accept parameters and instantiate valid representations of the data models. I intend to implement the front end data models and facilities using the angularjs framework.
A drag and drop interface on which the sources, destinations and log paths can be visualized, created and manipulated.
On this canvas, the objects that make up log paths will be visualized in a manner similar to network diagrams or class diagrams. The user will drag blank ( or pre-instantiated, like the system() driver) sources, destinations, filters etc from a sidebar onto the canvas area, where they will appear as manipulable objects.
When an object is added to the canvas, a popup window will appear with tool-tips to assist in instantiating the object model (such as prompts for driver type,required parameters, list of acceptable optional parameters, validation checking). The user can then can then create and drag connections between these objects, which when connected in a valid path ( source -> filter/rewrite rule/parser -> destination) will automatically cause a model for a log path to be instantiated.
To create this canvas interface, i will be utilizing the extensive and reputable mxgraph javascript library (see the draw.io for reference).
A user interface around the main canvas to hold tools and objects and options which do not need to be visualized. The user interface will also allow the user to search for functions or objects based on classification or their metadata.
Based on the meta data stored about the object types in the database, there will need to be front end factories responsible for accepting instantiated models represented on the canvas and sidebar, and converting them to actual config language snippets based on their contents.
This system makes it easy to construct config files but lacks the ability to directly read and represent existing files. I will need to either port or create a bison and flex parser in JavaScript to allow config files to be uploaded, parsed and visualize.
Interfacing with google drive would facilitate sharing config files and their visualizations conveniently.
- Familiarize myself further with the expectations of the web interface.
- Enlist the assistance of mentors in clearly defining the object, functions and relationships present in config language.
- Refine database schema through deliberations with mentor.
- Get familiar with the mxgraph library.
- Begin designing UI
- Create data model representations of the different objects the config language
- Populate database with models in order of importance
- Create simple rest api for interfacing with database
- Create simple front-end UI for adding and editing models in the database
- Continue to add models to the database
- Setup front-end web project
- Begin creating factories based on the different classes of objects in the database.
- Begin implementing the ability to instantiate objects based on the database models through these factories.
- Continue implementing factories with the ability to instantiate objects based off database models.
- Add specific validation checks and meta data models to be used later to generate tool-tips to help with the instantiation of models based on database models & relations
- Create canvas and data model that represents its state.
- Create api for rendering the various object models as diagrams on the canvas.
- Create facility to track the relationships of objects on the canvas
- Begin creating drag and drop interface for adding objects to canvas
- link the creation of objects on the canvas to the instantiation of the represented syslog-ng configuration data model.
- Create facility to monitor the relationships drawn between objects on the canvas and instantiate new data models to represent said relationships where necessary (create log path when sources and destinations are linked)
- Begin to improve aesthetics of the UI
- Create sidebar and other supporting UI elements to house tools and functionality
- Begin crafting popups and tool-tips to assist with instantiation of models added to canvas
- create generic system of extensive ui prompts and tooltips unique to each type of object to guide users to correctly instantiating them.
- refine UI design and workflow
- Create factories for each type of object that can accept an object of that type and based on the meta data in the database, convert that object into its textual representation as a snippet of config language code.
- create a ui element from which users can search for and instantiate global options.
- Continue improving and extending conversion to code snippets
- offer configuration file for download after converting the data models
- Continue to revise and improve on ui and functionality wherever it is most needed.
- Name: Radcliffe Robinson
- Github: github.com/Steams
- Email: radclifferobs@gmail.com
My name is Radcliffe Robinson. I am a second year computer science student at the University of the West Indies, Jamaica. I have personal and professional experience programming in a variety of contexts and languages. Last summer i worked together with a partner to develop a memory space and time crucial billing server in C, to validate, route, and bill calls coming through an Asterisk pbx system. This project imparted onto me knowledge of networking programming in C, multi-threading, and low level system facilities such as message queues and semaphores.
In the domain of web programming, i have developed many personal projects over the years, such as a beautiful note taking and sharing system and a voice controlled sign up and showcase app for my school's Computing society. Professionally, i have designed and developed a front-end interface for managing and billing customers using the aforementioned billing server and a resume comparison and job matching web system.
I have, for the past few years, been passionate about web development because i feel that, as an api for building user interfaces, it provides a level of structural freedom which allows us as developers to restructure or code base to best represent each unique solution as well as an unparalleled ease of implementing complex features. This has made the web my favorite programming interface, one which has consistently been able to scale with the programming ingenuity of its developers and to the demands of its users.
I first became interested interested in syslog-ng as a potential user while searching for a feature rich log processing solution for a billing system a partner and I built in C to accompany the Asterisk pbx system at our internship.
For me, this project represents an opportunity to further my progress towards the mastery of programming for the web environment and to exercise a level of my aesthetic competence and creative freedom which i have not had the opportunity to in a professional context. I am confident in my ability to plan out and implement the solution i have proposed and if given the opportunity, am looking forward to working together with the syslog-ng team this summer.