Skip to content

Organization

Jump to bottom
William Lallemand edited this page Oct 2, 2023 · 12 revisions

HAProxy Project Organization

Being a critical piece of infrastructure at many places, HAProxy must remain extremely robust, which mandates to follow a certain number of best practices all along its development cycle:

  • use of modern version control (Git), with one commit per change and one change per commit
  • changes are only accepted if they come with the accompanying documentation
  • code is split in branches:
    • development only happens in the development branch
    • maintenance (i.e. only bug fixes) happens in stable branches
    • multiple maintenance branches supported in parallel and spanning several years allow smooth transitions
    • too old branches are dropped when not used anymore: if no feedback comes from users after many years, these branches present a risk of getting faulty backports and it is preferred to officially recommend not to use them anymore.
  • strict non-breakage rule within stable branches: users must always have the ability to update or revert their deployments within the same branch without touching their configuration.
  • a large community of highly skilled users stress the code in various environments and with various setups, reporting issues, suggestions for improvements or patches.
  • development is open to public scrutiny and subject to code reviews. While public reviews are not systematic depending on who submits a change and what risks the change represents, developers are always accountable for their changes and will be engaged in troublshooting sessions that may be related to them.
  • detailed descriptions of the intent behind a change: each commit message describes and justifies the change, the decision taken, possible alternative choices that were considered and evicted, possible impacts, and instructions for those performing the backports to stable branches
  • code areas are split in subsystems, officially maintained by maintainers who are more knowledgeable and skilled on specific areas.
  • regression testing suite run by developers before committing changes (roughly one hundred of tests run by VTest)
  • continuous integration: committed changes are built and regression-tested on approximately a dozen of operating system, architecture, compiler, and library combinations
  • fuzzing: the code is executed in a fuzzing infrastructure (oss-fuzz) to potentially find corner cases that were not covered by regression tests.
  • code coverage: Coverity scans the code base and reported issues are triaged and addressed
  • issue tracker: bugs are discussed publicly so that multiple users can share the effort of bringing quality bug reports
  • debugging tools: the program includes a comprehensive set of introspection mechanisms and advanced diagnostic functions to allow users to watch for a certain event and/or provide accurate reports that limit the number of back-and-forth exchanges needed to figure what is going on.
  • security team: a small set of trusted maintainers handle security reports with top-most priority
  • security coordination: a wider list of trusted users, mostly distro maintainers and fast-reacting infrastructure operators, are notified upfront of upcoming fixes so they have the time to update their packages and test a fix before a critical issue is disclosed.
  • resilience: a handful of highly trusted developers have the permission to commit code to the project and to emit releases, and do it regularly so as to be certain nobody is critical
  • reminders: because we're all humans, we can fail to respond or forget something, and several automated tools periodically remind us we have certain things to handle.

Communication channels

Based on preferences, availability and habits, discussions can happen on multiple media.

Note: As a reminder, please do not ask for help nor send direct questions about haproxy to the persons listed in this document, they are unlikely to be able to help you and it may quickly result in your address being banned from all channels, and noone willing to respond to you anymore. Please only use the channels below.

Participants to discussions on the various channels are asked to keep in mind a few basic principles that improve communication and relations with others, and that probably relate to common sense for those used to community-driven projects:

  • "hello", "please", "thank you" etc are often expected as in any public place, and their absence may often cause some unneeded tensions
  • remember that each message posted is delivered to thousands of professionals and passionates, and that the mailing list's archives are public
  • do not reply to spammers nor send inappropriate contents
  • do not be needlessly provocative
  • do not quote private conversations without each participant's consent
  • do not include other participants in a conversation without making sure they're aware that the conversation is public
  • avoid top-posting in e-mail replies (though it might sometimes be acceptable when forwarding an integral message). At the very least, apologizing for doing so when it happens due to occasional constraints should limit the number of times it occurs.
  • trim unused parts of a message when responding to some topics, to save the readers' time. This is especially true when reviewing patches, where irrelevant parts should simply be removed to help others focus the relevant ones.
  • try to be gentle to each other and assume that misunderstandings and disagreements are most often a consequence of language barriers, as not everyone is fluent in English and some even don't routinely use a latin alphabet and might have to rely on a remapped keyboard to communicate in public places.
  • as much as possible, nick names or first names should use only characters that can be typed on a QWERTY US keyboard so that all participants can easily call any other one or just refer to them.
  • respect each others' work and valuable time: it is not appropriate to gratuitously criticize others' patches without having a better option to propose; do not send untested patches or solutions to problems unless explicitly mentionned (i.e. "I think this could work" is much different from the ambigous "Try this").
  • make sure a problem is well understood before proposing someone a solution; keep in mind that a failure in production can have dramatic consequences for those asking for help, who sometimes suffered such an event already before asking
  • try to adapt to the requester's skills when asking to provide more information on a problem. Sometimes operations require to wait for certain time frames and asking for unneeded information can postpone a problem's resolution
  • when a proposed solution works, please report it, as it's often rewarding for the person who devoted time to help, and it also helps others looking for the same solution
  • it is easy to miss a question. Some newcomers' questions might remain unanswered, and they might feel embarrassed to ask again; even when not having a solution to propose, responding just to "reload" the conversation, noticing that nobody responded can be much appreciated by them
  • remember that mistakes happen all the time (including regarding the principles above) and that replying with an excuse is a great way to terminate a discussion on a positive note which preserves one's image and that it is probably something that others expect anyway.

Mailing list

Originally there was only the mailing list to discuss development, help users, and report bugs. Nowadays there are more options to interact with the rest of the community, and depending on the type of discussion expected, the level of knowledge of the user and personal taste, participants will choose one or another. The mailing list remains the preferred way to reach maintainers, send patches, discuss changes and exchange arguments for or against a design choice. Many of the 1000+ participants are long term users who have the deepest experience that may be found on the project and such architectures in general. Many of them discretely run the most visited internet platforms and are skilled on way more components than just haproxy alone. This is how they can provide solid advices which are considered extremely valuable, and why their opinion is well respected among the community.

The mailing list is open for anyone to post; registration is not necessary (but in this case it is recommended to mention in the message that the poster is not subscribed and wants to be kept in CC). The list archives since end of 2008 can be consulted on The Mail Archive.

The mailing list does have a little bit of spam that slips through the anti-spam, whose tuning primarily focuses on not blocking valid messages. It does however reject HTML-only messages, which almost always comes from advertisers and are never welcome in public places. Attachments should be kept small (e.g. patches), and messages exceeding an unspecified size considered as abusive will silently be dropped (this never happens for valid uses). Images in general, such as screenshots (especially to display text, config or network captures), large e-mail signatures or those with advertising slogans, posts from pseudonyms instead of real names, and messages responded to using top-posting are not welcome and tend to result in some messages to be generally ignored by the members who find these practices impolite.

It is possible to test end-to-end connectivity, for example after creating mail filters or when suspecting a problem, by sending one empty message with an explicit subject such as "Test - please ignore" that will not distract other members.

The mailing list is part of the core infrastructure, and any issue (blocked message, service down or lost unsubscribe link) should be reported to their admins. It's important to note that a message posted there is instantly forwarded to more than one thousand destinations, some of which are themselves public archives, and cannot be deleted. The poster is entirely responsible for making sure not to accidently post sensitive information such as passwords or secret keys.

Discourse

In addition to the mailing list, is a web-based Discourse forum on https://discourse.haproxy.org/. This one is graciously hosted on the Discourse infrastructure. Discourse tends to be used more by newcomers and users seeking help, but will also see people who are more comfortable with web-based interfaces than e-mail. Developers are rarely seen there since such a format is distracting and not optimal for mass-processing, however they are occasionally invited into some discussions by other members. Due to the nature of many posts (ask for help), there is a tendency to see questions flowing from many people and responses coming from very few people. At least for the sake of offloading a bit those who devote their time to help others, it is kindly asked that participants who were once helped stay active and in turn participate in helping others (which fortunately, some do). The forum is operated by the following people:

IRC

An haproxy IRC channel is available on Libera.chat: irc://irc.libera.chat/%23haproxy. There doesn't seem to be publicly accessible archives. A bot keeps it synchronized with the Slack channel (see below). This channel is administered by:

Slack

There is also a Slack channel dedicated to haproxy on https://slack.haproxy.org/. However there are no publicly accessible archives, and access to the channel is restricted to users having an account only, which can be obtained by requesting an invite. It is probably safer than Discourse or the mailing list to exchange confidential information there but on the other hand, by not being indexable nor publicly searchable, these discussions do not benefit anybody else. As such this channel is more recommended for people seeking support on very specific points, while Discourse and the mailing list are more recommended for people seeking general guidance. This channel is administered by:

Security list

The right place to send issues and security reports is the public mailing list and the public issue tracker. However, a very small crew of maintainers and long-term members constitute the security team who will receive messages addressed to security@haproxy.org. This is a private list to be used exclusively to report critical security issues that must absolutely not be discussed anywhere else. The sole purpose of this list is to find an urgent remedy and get it committed and distributed to the usual packagers as fast as possible to limit end-user exposure to an imminent threat. This is the worst way of getting a bug fixed, as issues fixed in emergency by the wrong people often require incremental fixes after they are made public. For this reason, at any moment, the list members may decide to bring other maintainers in, or to switch to a public discussion if they estimate that the security risk is not that high or that the risk of breakage is too high. In addition, if a configuration-based remedy is found, usually the bug will be made public so that users are better prepared to address it. Reporters are asked to indicate upfront when they do not want to be credited as the bug reporters (and who should be credited for it if at all), as by default they will be credited for it. Under some exceptional circumstances, the members may accept to keep an issue under embargo for up to 7 calendar days after the report even if a fix is available, in order to leave some time to the reporter to test and deploy the fix. Slow internal processes affecting the reporter and opportunities to announce it at conferences are absolutely not accepted as valid reasons for requesting an embargo, since purposely keeping users exposed to a known bug is generally considered to be a very bad security practice. During this period, package maintainers, a few trusted users known for their ability to quickly test a fix on live traffic, and persons doing QA may also receive a copy of the fix, without the reporter having anything to say about it. Occasionally, package maintainers will ask for a few more days to get their packages ready, which can slightly extend the initially agreed upon embargo. Sometimes a package maintainer will request a CVE to ease their internal tracking, and if its ID arrives fast enough, it may be included in the commit message. Reproducers are usually not shared but may exceptionally be shared with trusted packaged maintainers who need to test their backports. Vulnerability reports must focus on accessibility first, be sent in plain text only (no PDF nor screenshots), without the need for external tools, account creation nor signing anything. It is preferred when the reporters provide a patch that seems to fix the issue for them. In case a reporter makes it too difficult to work on the resolution, the list members may unilateraly decide to make the issue public in order to increase the chances to find a good solution in reasonable time without the reporter's help.

General contribution process

The very first thing to do before trying to contribute anything is to read the contributer's guide. This is mandatory. Otherwise you'll risk to have your patches incorrectly split, sent for the wrong version, or an unacceptable commit message. It will then be obvious from your submission that you failed to do so, and as a result, nobody will be interested in reviewing your submission! So please do it, really!

Changes to existing parts have to go through these parts' maintainers, with the development list CCed. This list address is haproxy@formilux.org. The maintainers are listed in the file called MAINTAINERS at the root of the project's source. If no maintainer is listed, it's likely that the area in question does not have an official maintainer and is co-maintained by everyone, and in this case the change must be submitted for public review on the development list. Unless contributors were already actively discussing their changes with a maintainer and the maintainer took the decision to pick them, the list must be kept in copy. When developers submit additions resulting in a whole new subsystem, they will generally be asked to engage on maintaining it for the long term (either individually or their employer). There is nothing formal here of course but this can mean thinking twice before adding support for a fresh new protocol for example.

Newcomers are reminded that most of the current developers and maintainers are not english-native and that nobody must feel ashamed for their language difficulties on a public list whose members are benevolent and willing to help improve the submission quality and fluidity. The efforts are shared in good faith between participants. Contributors are kindly asked to make reasonable efforts, and in turn maintainers are asked to take care of silently addressing some small mistakes for the sake of better quality.

Maintainers will collect changes for their subsystem and, at their discretion, either ask a committer to take them immediately, or prefer to keep them in their own branch. In the latter case they are encouraged to keep that branch public to avoid receiving multiple submissions for the same change, as this is particularly annoying for a contributor, especially when it happens for a first submission. Fixes for bugs should not be left rotting for too long. Either they are valid and they must be merged, or there are doubts and they must be discussed.

Committers can apply changes to any area. Normally and by principle, they will submit their changes to the relevant maintainers for review, or will discuss with them, but if a maintainer does not respond, or if they feel confident enough in a change that is of low enough importance or needs to be urgently applied, they are entitled to commit it. While it may sound a bit impolite at first, it is crucial in order to efficiently address bugs that might affect lots of users. Committers are aware of the risk of breaking working stuff and do endorse this responsibility, and by doing so, commit to helping maintainers resolve any trouble they might have caused (including by reverting their change if it was wrong).

So a change contributed by a newcomer will often flow through 2 different people, the maintainer, then the committer, and in some cases, the committer only. Note that each of them may occasionally apply tiny changes to the patch's description, documentation or comments in order to fix aesthetic or spelling mistakes, and more rarely to fix a bug that was discussed during the submission. However such modifications will not be performed without the submitter's or maintainer's consent if any of them has signed the submission by adding a Signed-off-By tag at the end of the commit message, at the expense of more back-and-forth exchanges. This means that in case of trouble, the first person to report the problem to is always the first signer of the patch, and otherwise the committer. The committer of a patch can be listed with git log --format=fuller <commit_id>.

A self-organized community

There are fundamental differences between a contributor, a maintainer and a committer:

  • a contributor may be anyone, an individual, or a company's employee or a company's development team. At least contributors must be identifiable so that in case their change cause problems, they can be contacted to give their opinion. But this is only for the sake of being polite, as any maintainer may unilateraly decide to revert a change considered wrong or to adjust it again. Obviously, contributing code takes time, and nowadays most of the code is contributed by developers working for an employer whose activities benefit from haproxy being improved.
  • maintainers may also be anyone, an individual or a company's employee, but their status demands that their role is secured over the long term and at least for the whole duration of the branch that follows their last important contributions. For an individual, this may mean that the person is willing to devote some spare time to the project. For a company, this would rather mean that several people are identified as co-maintainers so as to cover the situation where an employee would switch jobs. As a side note for individuals, there are plenty of companies willing to hire maintainers, so working on week-ends is not the only option.
  • committers are exclusively individual developers who have a personal interest in the project and have developed a longterm trust relationship with the other members to the point where others considered it beneficial to invite them. These ones know that their role is crucial to maintain a high responsiveness to change requests from maintainers. Usually they are the ones spending the most time chasing bugs. They also help cover each others' vacation or lack of time. It is not uncommon to see them merge code on evenings and week-ends even though this must not be the rule. This level of involvement explains why there are so few and why a few others have already declined. For this reason, some of them will often work for companies which are willing to directly contribute to the project by reserving some time for their activities.

The current list of committers is:

Tools and services

All these best practices involve a large number of efforts that cannot be carried by developers alone, and that will often surpass their skills in certain areas. For this reason, the project relies on a wide number of external tools and services, often proposed by the people managing them based on availability and preferences, sometimes based on generous life-time donations from some companies willing to contribute to the project.

All the efforts are distributed over a number of areas:

  • core infrastructure services (git, www, mailing list)
  • development tools (Github issue tracker, CI, code coverage, fuzzing, spelling)
  • automated bots (doc regeneration, PR bot, stable bot, bugs bot, nightly snapshots)
  • system-specific packaging (distros, OS vendors, docker images, ...)
  • communication channels (Mailing list, Discourse, IRC, slack, security list)

For various reasons ranging from account mappings only allowing service owners, ability to access or provide certain services, historic participation to the project, or availability, the people doing most of the work on certain services are not always the same as those having full administrative permissions. Below, project owners will be listed as owners. They are not necessarily very active on the service but must be involved from time to time to perform one rare operation requiring their approval when the admin cannot. Administrators (who can manage accesses and permissions) will be listed as admins and those doing the actual work will be listed as operators. In practice admins should not be bothered unless there is an abnormal difficulty to access the service. Sometimes an admin is listed as a backup admin, meaning just that this is not the right person to contact unless all others are unreachable. As a general case, the contacts listed here are those who are able to diagnose and operate their respective services, and under no circumstances they should be bothered with haproxy-specific support.

Core infrastructure

The core infrastructure historically runs on the formilux.org servers. This comprises the main git repositories (http://git.haproxy.org/), the web pages reporting the project's status and current versions (http://www.haproxy.org/), the download area, and the mailing list. These servers are operated on spare time by 2 persons, to be contacted in this order in case of issues (service down, corrupted file, unsubscribe from mailing list) :

Development tools

GitHub

GitHub is used to coordinate development with a number of useful development tools. As such, only the development version is found there. Stable versions are not hosted there at the time of writing for several reasons that may be addressed in the future.

The haproxy organization on GitHub has one owner and two admins who have all rights on the repositories, who can add/ban members, forcefully delete an issue which accidently leaked sensitive information, and various other highly sensitive operations that fortunately happen less than once a year. If you face such a difficulty (and only in this case), please contact, in this order:

GitHub's issue tracker is used to track issues, feature requests, and development notes. This issue tracker is reasonably flexible, easy to use and forgiving, so it provides some long-term benefits at a limited extra cost. The issue tracker is operated and maintained in a clean state by the following volunteers who will (as time permits) classify issues, assign them to project members, close obsolete/incomplete/invalid ones, create new labels, ask for complements of information when questions are not clear, or sometimes try to reproduce an issue:

Continuous Integration

GitHub is also connected with two CI (Continuous Integration) suites, one being Travis (https://travis.com/haproxy/haproxy/) and the other being Cirrus (https://cirrus-ci.com/github/haproxy). These suites periodically build haproxy on a wide set of operating systems, architectures, compilers and options combinations, then execute it under the regression testing suite, and report failures if any. These two CI suites are configured and operated by:

  • Ilya Shipitsin chipitsine@gmail.com (operator)
  • haproxy organization's owners for admin stuff (normally not needed)

Note: Please be kind with Ilya, as certain constraints on the CI configuration process imply that any configuration change he does first needs to be merged into the main haproxy repository by a committer before his changes are effective, so this necessarily adds a certain incompressible delay.

Coverity

A code coverage suite, Coverity, periodically runs on latest code and reports suspicions of bugs. This requires a lot of care and manual triage to eliminate obvious false positives and tag reports based on developer feedback. This suite is run by the following people:

Fuzzing

Google provides an OSS-Fuzz platform that is used to stress some code areas that are not easily tested by humans or by regression testing. Some interfaces have to be explicitly written in order to expose some haproxy areas to this platform. Problem reports are publicly available for anyone to participate (based on haproxy project owners' request). This service is operated by the following persons:

Spelling

A periodic check is run on the whole code and documentation for spelling mistakes based on GitHub Actions configured within the haproxy project repository itself. Based on the reports, patches are created and submitted to the mailing list, to be applied after a quick review. Usually the corrections are valid. This service is operated by the following people:

Automated bots

A number of tools have been created to help developers, to provide up to date contents, and to send reminders.

Documentation generator

The up-to-date documentation for all maintained versions, a few unmaintained ones, as well as latest development snapshots, is processed daily by a documentation converter called haproxy-dconv which pretty-prints the documentation to render nicely in a web browser and make it easily navigable. This tool is operated by its author, who may be contacted if some parts do not render well or if for whatever reason the bot seems not to be running anymore :

PR Bot

History has shown that pull requests sent via GitHub almost always are of extremely poor quality and do not respect the coding style not contributing guide. They almost always require quite some extra processing and reviews, which adds too much burden on a small crew. Instead they are automatically closed after having been forwarded to the mailing list prefixed with [PR] where many more participants can see them and comment inline, and where a maintainer can easily pick the patch and clean it up before merging it. This is handled by an automated bot developed and operated by its author, who may be contacted in case the bot would not work anymore, resulting in pull requests remaining pending in the project:

Stable-bot

Developers usually do not see time fly, and by regularly postponing a maintenance release to include a fix for the last known bug, they can easily end up not producing a single maintenance release for several weeks, sometimes months. In order to limit the risk of such drift, a reminder is sent once a week on the mailing list prefixed with stable-bot:, indicating for each actively maintained branch the list of pending fixes, and an estimate of what ought to be an optimal release date based on the severity of each version's known bugs. This tends to encourage developers to focus a bit more on lagging backports, and users to pressure the developers to create a new release. This bot is developed and operated by its author, who may be contacted to suggest adaptations, or if no stable-bot messages are seen for several weeks on the mailing list:

Nightly snapshots

Every night, snapshots of the latest version of the currently maintained branches (including the development branch) are produced in the snapshots sub-directory of each branch's source (e.g. http://www.haproxy.org/download/2.2/src/snapshot/). These snapshots include a tarball of the latest version of the code, a tarball of all patches since the last released version, and a change log from the last released version. The snapshot generation belongs to the core infrastructure and is operated by the same admins, who need to be contacted if snapshots fail to be produced.

Bugs bot

In addition to the snapshots above, every night, the list of known bugs affecting all versions of all maintained (and some unmaintained) branches is updated on http://www.haproxy.org/bugs/. The bot belongs to the core infrastructure and are operated by the same admins, who need to be contacted if branches are missing or not updated anymore.

Packaging

packages

Clone this wiki locally