Skip to content

GSoC2016 Proposal: Automated release generation for syslog ng (Asad)

Jump to bottom
Asadullah Hussain edited this page Mar 23, 2016 · 14 revisions

Summary:

The idea is to develop a Python module that creates a new release of syslog-ng. Apart from pushing this release via Github, the module will also automate tasks like Building the source via OBS, Generation of Packages (Debian, Fedora) and their testing.

Motivation:

syslog-ng is a widely used tool with multiple contributors. As with every organization the main motivation is to deliver releases more often and reliably. Manual release generation and deployment is the major hurdle in achieving this goal. It is also prone to error and unreliability because different users may interpret release instructions differently, leading to non-uniform release format.

DevOps is a concept that has been employed to automate software deployments and we will be deploying relevant DevOps concepts (release automation to be specific) to syslog-ng as well. The core functionality will be implemented in a Python module with the possibility of utilizing Atlassian tools for uniformity and hardening.

Project Breakdown:

Currently the syslog-ng release process consists of two parts i.e., "Preparing Release on Github" and "Building Packages on OBS". In the following section the breakdown of each part is given. Our Python module will be utlizing Github API & OBS API to automate these steps:

Prepare Release on Github:

This involves creating a new ("release")[https://help.github.com/articles/creating-releases/] on Github. My proposal is to utilize the Github API to automate the release steps.

Python has very handy libraries for using the Github API. But when needed we also have the option of invoking API calls directly via requests or urllib modules of python. This is just an overview of API mapping to our release steps:

API Repositories: These function calls will help us implement the following steps:

  • Increase the version number: This can be achieved by the API call Block Releases. Version can be incremented by getting the release number through "Get Release" and then incrementing it and pushing the new version via "Create Release".
  • Generate changelog (collect contributors, closed issues, commit messages, and so on): Changelog generation is also possible by parsing data received from corresponding API calls for collaborators , commits , issues etc.

The tricky part in this step would be to sort out info from the text dump returned by the API call. An overview of the API shows that all info is accessible but the challenge is in parsing this info a generating an meaningful log.

  • Tag syslog-ng release in GitHub: Fetch the "git tag" from source and Tagging done by "create release"
  • Create syslog-ng distribution tarball Python has a library for creating tarballs
  • Prepare a release draft in GitHub: The API has a function to generate a draft.

Build:

Before publishing the release, we have to verify that it builds properly. For this purpose openSUSE Build Service (OBS) is utilized. OBS is an opensource online build system that allows users to build packages for different platforms. OBS can be used via Webclient, Command Line Tool- OSC or REST API using curl. Our python module will be utilizing the OBS API to build our source using simple POST/GET functions.

NOTE: We can also use a localized Jenkins instance to build our source.

For syslog-ng release we perform the following steps on OBS:

  • Upload distribution tarball to OBS project: The API has corresponding function to upload package files e.g., PUT /source/<project>/<package>/<filename
  • Publish Release: The build results from OBS can be collected by using the following function: GET /result/<project>/<platform>/result.
  • Generate Packages: Our python module will also provide templating of .spec/.dsc files for generating rpm/debian packages.

Once these results are correct the release will be published on Github. The target build will also generate standalone Debian and RPM packages which are also accessible via "GET". Hence completing the release.

About Me:

  • Hi, Asad here and I am pursuing Master of Informatik at Technical University Munich.
  • I have extensive experience in Software Development (C, Python) & Systems engineering. I've worked for 2 years on projects by Silicon Valley companies such as FireEye, Cavium, Dell and Intel from the platform of Ebryx & xFlow Research.
  • I successfully participated in GSoC 2015 under the CentOS Project, where I developed an OpenStack Addon for Anaconda (redhat/fedora/centos installer) in Python.
  • I have been working as a DevOps Engineer at T-Systems, Germany since last October. My team is developing a continuous integration pipeline by automating infrastructure and providing it as a service (IaaS) for development, test and operation. I have implemented release automation infrastructure via Puppet and this is why this project is so eye-catching for me. As I have experience of release automation pipeline by integrating tools like Git, Jenkins, Jira, Vagrant.

The challenge of developing this pipeline for syslog-ng via Python is very exciting and my aim is to not only develop a Python module that executes release automation steps but to also enhance this process by automating the entire Continuous Delivery (CD) Infrastructure (Infrastructure as a code). The end goal is to create a Git repo that props up a simple "Box" (docker/vagrant/qemu) and executes the release automation inside this box from Git Checkout till mailing list emails.

Timeline (tentative):

My Proposed Timeline is as follows:

  • (March 2016) Familiarization with syslog-ng release automation .

    Deliverable: Ability to manually release syslog-ng, initiated by pull request till package generation on OBS.

  • (April 1-15) Familiarization with OBS Infrastructure and compare if a localized Jenkins is better.

    Deliverable: OBS feature list comparison with Jenkins build.

  • (April 15-30) Familiarization with Git API & OBS API.

    Deliverable: API Function Mapping to Each Release Step.

  • (May 01-31) Development of Prepare Release class

    Deliverable: Fully automated GitHub part of release. Mostly text parsing.

  • (June 01-30) Development of OBS Build class

    Deliverable: Fully automated release module.

  • (July 1-7) Integration with syslog-ng CI /Bug Fixing

  • (July 7- 20) Development of standalone box for Testing

    Deliverable: A script that launches multiple Vagrant boxes which will build/test syslog-ng in different platform/OSes.

Clone this wiki locally