[GSoC'16] Automated release generation for Syslog ng

Ankush Sharma edited this page Jul 8, 2017 · 7 revisions

#Abstract

Syslog-ng is a famous log management software with content-based filtering, rich filtering capabilities, flexible configuration options, TCP transport and many other features. The project aims to implement an automated release generation tool for the Syslog-ng project considering the current release requirements of the project.

#Why automated release generation is required? Automated release generation is an agile software development practice which brings the following benefits:

  • Manual release generation will require developer to have proper skill set while automated release generation requires less knowledge.
  • Manual release generation is more error-prone as compared to automated release generation in which once the software system is set up with required configurations, have very less chances of errors.
  • Frequent releases can be done.
  • Developers get to spend more time on developing.

So, it would be of great worth if Syslog-ng has such release generation software.

#Detailed Description: I am proposing to divide the project into two components as per usability:

  • Command line tool
  • Web interface

A command line automated release tool will do the job, but it would be nice to have a web interface to ease up things.

a) The outer work flow

Overall the process for release generation for syslog-ng can be visualised by this:

Workflow

b) Architectural requirements

Overall, I propose to break the development into three parts:

  • Github API client :- As per the requirements, we need to write a wrapper library across the REST API offered by GitHub. We cannot hard-code the REST API endpoints into our code because it is a bad practice as the endpoints may change later on. Wrapping the API into a library will provide easy backwards compatibility in case changes are made to GitHub's REST API in future. So, a module representing the GitHub client will be part of the architecture.

  • OBS API client :- In a similar manner, it will be required to have a OBS client module that will make call to the Open Build Service.

  • Core :- The overall task of release generation can be divided into the following three sub-tasks:

  1. PreRelease
  2. TestRelease
  3. PublishRelease

c) Core Architecture

The inner details of the three sub-tasks can be encapsulated into three Classes as mentioned below:

1) PreRelease : It encapsulates the functionality required to prepare the release, i.e the bootstrapping process. It is required to create a release branch from the master branch and generate a distribution tarball from the thus made release branch. All such functionality will be provided by this class.

2) TestRelease : This class will be responsible to provide methods through which we can upload and test the previously created distribution tarball by making use of the OBS.

3) PublishRelease : Now, finally if the OBS builds get successful, we can generate the change-log, tag the release in GitHub and add release notes to it. The release publishing functionality is offered by this class.

A UML diagram showing a brief view of the architecture is shown below:

classes-uml

Note: Only some of the methods are shown to give an idea of distribution of work.

d) The Flow and Usage

Simply, the flow would be like this :

                    prepare_release => test_release => publish_release

i) Let's go command line
Overall, the developed tool can be invoked through the command line to carry out the process of automatic release generation. But it would be nice to have the functionality to carry them independently. For eg. say autorel is the name of the developed tool :smile:

# Just test the previously generated tarball
$ autorel --test

# Woot! it works out fine, now publish the release
$ autorel --publish

ii) Web interface, why ?

  • OBS build logs and previous release attempts logs can be saved and shown in the web interface.
  • Change-log edition can be done by providing a markdown editor.
  • Project configuration for eg. API keys etc. can be edited through the web interface itself in an easy manner.
  • A command line tool is indeed a command line tool, how much verbose it can be! So, enhanced/real-time logging can be supported via a web interface.

#Technology requirements

  • Python3 would be better choice instead of Python2.
  • A microweb framework like Flask or Django can be used to develop the backend of the web interface.
  • For the development of the two REST clients, a HTTP/HTTPS request library is required. The best choice would be indeed the famous requests library. Apart from this, any other specific modules can be chosen upon discussion with the mentors.

#Code quality

  • Legitimate code : A good idea would be to use code linting tools such as pylint, pyflake etc to keep up with the code quality and enforce the coding standards such as PEP8. These are community driven tools and are being used in the projects like Xen, Mercurial etc.
  • Testing : Writing unit tests is as important as writing the development code. It is required to make use of different testing methodologies like stubbing, mocking etc. Since, we will be making the API requests and database requests, so mocking of these requests is required to be done. We need to generate fixtures so as to write unit tests harnessing them. Two good library choices for mocking requests are:
  1. Httmock
  2. Responses
    Later on code coverage analysis can be done using suitable tools.

#Timeline I will graduate next month. So, I will be free in the summers and can easily promise > 40 hours per week. I would like to take the following approach:

  • 22/4/2016 - 22/5/2016 (Community Bonding Period):
    • Go through GitHub API and identify the REST endpoints of interest.
    • Plan on the front-end of the web interface.
    • Research about Debian and RPM packaging.
    • Set up development environment with the selected toolkit.
  • 23/5/2016 - 5/6/2016 (Actually coding period starts):
    • Code the GitHub API client.
    • Write test suit in parallel.
  • 6/6/2016 - 21/6/2016 (Preparing for Mid term evaluations start):
    • Code the PreRelease, TestRelease and PublishRelease classes.
    • Write test suit in parallel.
  • 27/6/2016 - 17/7/2016:
    • Code the OBS API client.
    • Integrate all the coded modules.
    • Finish the command line tool.
    • Start working on the web interface.
  • 18/7/2016 - 16/8/2016 (Preparing for end term evaluations):
    • Finish the web interface part.
    • Work on RPM and Debian packaging.
    • Code refactoring and developer docs generation using inline code commments.
    • Integrate all the elements together.
    • Increase code coverage by writing more tests.
  • 25/8/2016 - afterwards (Optional):
    • Work on experimental features.

#Deliverables The final deliverables list would be like this:

  • We will have the following under our toolbelt:
    • A self sufficient command line program that can do release automation
    • A web interface that can be used for configuration,logging and other useful stuff.
  • As a demo, the developed tool will be able to carry out the release automation process of a dummy repository.
  • Debian and RPM packaging support within the tool.
  • Developer documentation for the developed tool and it's configuration options.

#Why Syslog-ng ? I found out Syslog-ng two months back. I jumped into the source code and found it to be a great system to learn C, OS concepts, Linux tools architecture etc. As more of a Web Developer, I have never been into such Linux tools internals but I found the project of release automation of Syslog-ng to be a descent side helper project to start hacking on core Linux tools like Syslog-ng. I plan to learn more on the architecture of Syslog-ng and contribute code to it.

#About Me I am Ankush Sharma, a final year undergraduate at IIT-BHU, Varanasi, India. I mostly develop web stuff but have a knack of System Programming too. I love open source software and is a regular GitHub contributor. Here is some of my work:

  • Authored mailman-client.js as part of GSoC'15 for GNU-Mailman.
  • Developed Ptop, a task manager for Linux based systems. The project trended on GitHub for a week.
  • Developed Makemymails, a Django app for automatic personalized mass mailing which was runner-up for Gooole Cloud Developer Challenge 2013, India zone.
  • Delivered a talk at PyCon( National Conference of Python Programming Language ), India 2014 on the tools required to build Human Computer Interaction projects with Python. Talk link

Apart from this, I have many projects in Python, JavaScript, Node etc. Personal information:

Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.