A healthy open source community is one that demonstrates open practices, uses open infrastructure, and cultivates an open culture with the goal of becoming more sustainable. But even for the most seasoned community managers and community architects, measuring an open source community’s health is a complex, difficult, and occasionally intimidating task.
That’s because any picture of project health is actually a mosaic; multiple factors combine to depict a community’s overall health. You’ll never find just one indicator you can point to and say, "See? The community is healthy."
What’s more, the factors that determine overall project and community health are always changing. In the past, for example, some may have pointed to "total number of download" as a health metric. But today we understand that even millions of downloads does not reflect the fact that a communuity’s production and quality assurance processes might be out of sync—or that trolls are misbehaving on the project’s mailing lists. A more complete picture would consider not only results or outcomes (like downloads, release cadence, etc.), but also processes (onboarding barriers, how easily a project can discover and promote maintainers, etc.).
This chapter explains in more detail the importance of a metric-driven approach to building, maintaining, and growing an open source community. It outlines a few important factors to consider when assessing an open source project’s and community’s overall health. And while every commuity is different—and therefore requires different metrics for measuring success—this chapter offers a few example metrics you might consider when measuring your own community’s successes.
Establishing and maintaining community metrics is important for several reasons, including:
-
Providing objective measures
-
Promoting a culture of transparency
-
Encouraging community participation
-
Identifying potential community bottlenecks
Let’s explore each of these in more detail.
For any organization, you want to have a set of objective measurement to help you understand how you are performing. For businesses, it could be things like revenue, new customer aquisitions, costs, etc. that you can look at periodically (e.g., quarter-over-quarter) to see how you are progressing. For open source projects, it is also helpful to have a set of metrics that can help measure the health of your community. These could include things like number of contributors, how long it takes to close bugs/issues, responsiveness of reviewers, contributor diversity, etc.
Transparency is a key ingredient in all open source projects, so it is important to have insight into all contributors to your project. This isn’t necessarily to rank or highlight the most active contributors. Rather, this is to get a better sense of where the contributions are coming from (e.g., different geography, organizations, etc.). When you start going beyond 20-30 community members, it starts to become more challenging to have a good insight into all contributors if you don’t have a good set of metrics.
Community metrics will not necessarily provide a full picture of the community, but they do offer important insight into the state of the community. For example, metrics can show if there are regular activities on project’s code, bugs/issues, documentation, etc. for the project. Also, when people join a new community, they typically struggle with identifying and connecting with more experienced members. If the metrics dashboard includes a list of active documentation contributors, that could be useful for anyone interested in contributing to project documentation. Some communities even have a dashboard of first time contributors to highlight new community members. These may seem like small things, but they could go a long way to help new community members onboard into a project and feel welcome.
As open source projects grow, you will typically face challenges with responsiveness to community members. For example, code reviews may take longer or you may see a growing backlog of issues. If you have a good set of metrics (e.g., average time to close issues, median time for code review, etc.) that you review on a regular basis, they could serve as early warning signs before you start to have a number of dissatisfied community members.
Before you determine how you will measure project and community health, you’ll need to determine what you’re going to measure. You might begin by looking at:
-
Project life cycle
-
Intended audience
-
Governance
-
Project leaders
-
Release manager and process
-
Goals and roadmap
-
Onboarding processes
-
Internal communication
-
Outreach
-
Awareness
-
Ecosystem
Let’s examine each one of these in more detail.
An open source project’s life cycle affects many other considerations about a project’s health. Understanding the project’s place in that life cycle will help you contextualize your assessment (e.g., single-person governance can be common when a project is young—but less effective when a project is more mature). Monitoring contributor trends might reveal critical information about a project’s short-term or long-term future.
When assessing the project’s life cycle, consider questions like:
-
When was the project founded, and how old is it?
-
How often do new contributors join the project?
-
How frequently does the project accept new contributions?
Well-run open source projects demonstrate a clear understanding of the users (and contributors) they hope to assist and engage. When assessing a project’s intended audience, consider questions like:
-
Has the project clearly identified a intended audience, and who is it?
-
Is the intended audience the most appropriate one for this project?
-
Does the intended audience engage with competing or complementary projects?
Governance refers to the rules and customs that define who does what in an open source project and how they are supposed to do it. Healthy projects entail thoroughly documented (and continuously evolving) governance models. When assessing a project’s governance, consider questions like:
-
What is the project’s governance model, and is it publicly documented?
-
Does the model account for both technical and business concerns?
-
How do project members make and enforce decisions?
Refer to this guidebook’s chapter on governance for more consideration of this topic.
In healthy projects, leaders are visible and easily identifiable. Leaders often coordinate project work and establish a project’s vision, and usually have extensive knowledge of project history. When assessing a project’s leadership, consider questions like:
-
Who are the project leaders?
-
What are the project leaders' responsibilities, and are they focused more on engineering, marketing, or some combination of both?
In healthy projects, members have formally documented release processes and identified release managers to supervise those processes. When assessing a project’s release manager and process, consider questions like:
-
Is the project’s release process documented?
-
Does the project have an identified release manager?
-
How often do project release updates occur?
-
Do project releases occur on a steady and predictable schedule?
Healthy open source projects have publicly shared goals and clear processes for reaching those goals. Goals are attainable and clear deadlines exist for tracking progress toward those goals. When assessing a project’s goals and roadmap, consider questions like:
-
Are project goals clear and public?
-
Does the project have a clearly communicated process, and is it also public?
-
Do project participants have a history of meeting project deadlines?
New contributors are vital to project innovation and success. Healthy projects feature clear, welcoming onboarding materials that assist newcomers who wish to participate in the project. When assessing a project’s onboarding processes, consider questions like:
-
Does documentation explain precisely what the project is and how to use it?
-
Does documentation help new contributors get involved in the project?
-
Does the project accept contributions of more than one type (e.g., development, marketing, project management, event planning)?
Communication channels are key indicators of project health, as are a project’s internal communication practices. Issues affecting community health often emerge first in internal channels—such as mailing lists or chat platforms—where contributors and users interact. When assessing a project’s internal communication, consider questions like:
-
Does the project have sufficient communication channels?
-
Can people find and use these channels effectively?
-
Are channels regularly moderated?
-
Is channel communication governed by a code of conduct?
Refer to this guidebook’s chapter on communications norms for more consideration of this topic.
Outreach is the process of actively promoting a project and making others aware of it. Communities use written materials (e.g., social media, blogs, whitepapers), events (e.g., meetups, conventions), and educational tactics (e.g., demos, training sessions) for outreach. Healthy projects have adequate energy and resources devoted to outreach. When assessing a project’s outreach efforts, consider questions like:
-
Does the community use clear and consistent methods for outreach? If not, does it plan to establish a set of outreach methods?
-
Are people writing, talking about, and promoting this project and its technologies?
The project’s intended audience must be aware of the project and understand the problems it solves. Awareness is a desired outcome of a project’s outreach efforts and can be measured through user and contributor surveys or general marketing analyses. When assessing a project’s awareness, consider questions like:
-
Is the intended audience aware of the project?
-
Can people in the intended audience explain the project’s uses, features, and advantages over alternatives?
-
Do others working in an industry that would benefit from the project know the project exists?
No project exists in a vacuum. Projects frequently depend on one another. In some cases, similar projects can be competing to reach the same intended audiences. A community’s interactions with other projects in its ecosystem reflect the project’s health. When assessing a project’s ecosystem, consider questions like:
-
What are the project’s dependencies and what projects depend on it?
-
Is the community sufficiently integrated into the overall project ecosystem, intended industry, and organizations that may use the project?
-
Do members of that ecosystem view this project favorably?
Because no two open source communities are the same, every community will naturally have its own set of metrics for measuring health and success. Many factors can influence a community’s choice of metrics, but one of the most important influences is the community’s goals.
Some communities, for example, prioritize how quickly they’re able to merge code—so they track metrics related to this ability. But other communities consist of users and contributors working in heavily regulated industries (like energy or health care), where necessity dictates that decisions to merge new fixes and features take a longer time. Those communities probably wouldn’t emphazize speed of code review as much as, for example, code review efficiency.
Depending on its goals, a community might establish metrics for measuring things like work backlog, contributor diversity (e.g., organizational, geographical, etc.), flow of first-time contributors, localization and internationalization, or popularity of discussion topics. Communities should always establish metrics collaboratively and agree on them collectively. Hearing from a diverse group of community members is important for ensuring the metrics are inclusive and not just focusing on the work of a subset of the community.
It’s also a good practice to review the metrics periodically with the community and discuss if any adjustments are needed for your metrics. Even within the same community, you will likely need to evolve your metrics along with the community as your needs change.
In the past, many people wrote complex scripts/queries to get metrics for their communities. Nowadays, most of the software tools (code repositories, forums, issue trackers, wiki’s, etc.) that open source projects typically use have APIs, plug-ins, or even built-in dashboards that makes it easy to collect data for your community. So if you use tools like Discourse, GitHub, GitLab, Jira, or others, you may be able to save a lot of time by reading their documentations prior to implementing a new set of metrics from scratch.
Not surprisingly, there’s an open source project that is focused on community metrics. The project is called CHAOSS (Community Health Analytics for Open Source software) and it has community members from academia, companies that participate in many open source projects, open source foundations, and others. If you visit the CHAOSS website, you will find details on metrics across different categories plus implementation examples for many of these metrics.
If you browse through CHAOSS metrics, you will likely find plenty of metrics (and implementations) that will be applicable to your community. If you have an idea for new metrics that are not yet in CHAOSS, you can also start a discussion on new metrics in the CHAOSS community.
A number of open source communities have good documentation/code for their community dashboards that many other communities can take advantage of.
Many readers may be familiar with the CNCF dashboard and you can find details on their devstats project for their dashboard in the CNCF’s repo at https://github.com/cncf/devstats.
Another good example is the Ruby community dashboard and their FAQ page, which provides good insight into why they developed the dashboard and some of their implementation decisions.
Some communities publish contribution metrics after each release. Here is a good example from WordPress after their recent release, where you will see a lot of good visualizations for where the contrbutions are coming from.
Metrics are certainly important, but there are definitely shortcomings that we need to be aware of.
-
People often measure the most easily measurable in their metrics: this is human nature as we all want to do what’s easier. However, you run the risk of neglecting important aspects of the community if you only focus on easily mesurable metrics in your community. For example, it’s often easier to focus on inputs (e.g., number of commits/merge requests/pull requests) compared to outputs (e.g., the impact of a commit/merge request/pull request). Needless to say, ignoring outputs from the community will provide an incomplete picture of the community.
-
Over-reliance on metrics will provide an incomplete insight: No set of metrics will provide a full picture of communities (or any organizations for that matter). Although it is important to have a standardized set of metrics so that you can gauge your community’s progress over time, there will always be things that are extremely difficult to measure or quantify. For example, we all want contributors to feel a strong sense of belonging in the community and enjoy collaborating with other community members. Whether this is happening or not would be difficult to quantify, but you still want to have a good sense on this aspect of community health even if it requires other means besides metrics collections.
-
Ignoring intrinsic motivation: people often join (especially volunteer) organizations because they are intrincically motivated. For example, they strongly identify with group’s mission or enjoy a sense of belonging with other members. If there is too much emphasis on people’s contribution (or input) in metrics, you run the risk of losing sight of why people joined the organization in the first place. Most contributors in open source communities are volunteers who contribute in their own time, so ignoring their intrinsic motivation can often lead to negative consequences in the community.
Refer to the chapter on participant motivations for more on the topic of intrinsic motivations.
Beyond these shortcomings of metrics in general, the following are particularly relevant to open source communities.
-
Gaming contribution metrics: this usually happens when metrics are used as a main (or even a sole) basis for recognition in the community. Not suprisingly you will see behaviors like people submitting multiple commits/merge requests/pull requests for trvial changes when they could have accomplished the same thing with a single commit/merge request/pull request.
-
Vanity metrics: you also see vanity metrics outside of open source. A good example is placing too much emphasis on things like the number of social media followers. As in social media, quantity isn’t everything. Also, if you want to ensure that community members' intrinsic motivation is satisfied in the community, vanity metrics is definitely not a good way to go.
-
Making comparisons between different open source communities based on a few metrics: sometimes you will hear things like, "Project A had more than 5,000 attendees in their last conference," or, "Project B has 1,000 contributors," and people have similar intentions for their community. Before you are tempted to compare your community to others, it’s important to consider if you are making apples-to-apples, that is, a like-for-like comparison. You may be in a different industry, in different stages of project maturity, or have a different scope, etc. and a direct comparison may not be appropriate. Before you think about wanting 1,000 contributors in your community, you may want to ask basic questions like do you really need 1,000 people to accomplish your project’s goals?
-
Too much focus on code contributions: it may be because there are more tools available to capture code activities, but there’s a tendency to focus mostly on code contributions in open source communities. However, it is important to remember other valuable contributions such as answering questions on forums, triaging issues, maintaining wiki pages, etc. There should be an effort to ensure that community metrics reflect a variety of contributions (both code and non-code) so that no one in the community feels left out.
Finally, here are some dos and don’ts when you are working with open source community metrics.
-
Do make metrics public: This may be stating the obvious, but transparency in open source should also extend to metrics. When you develop metrics, it helps to include a diverse group of people in the process so that metrics are inclusive and consider all contributions. Also, if any adjustments need to be made for your community metrics, it’s likely that we will first get that feedback/suggestion from community members. Also, all metrics should be open to everyone so there is confidence in the data.
-
Do use metrics for spotting outliers: metrics are particularly useful for highlighting areas that aren’t doing well. Good examples are things related to throughput such as time it takes for close issues, forum posts to be answered, code review, etc. In these examples, metrics are a great tool that can help identify potential bottlenecks early.
-
Do use metrics as a starting point for gaining further insight into community health: Metrics may tell you what is happening in your community, but you typically will not know the why just by looking at the numbers. If the metrics shows that the number of first time contributors are declining, you will probably need to have some one-to-one hallway or phone conversations in order to identify the causes of the decline. Metrics will highlight the symptoms as a starting point, but people will then have to do the work from there.
-
Don’t use metrics as a sole basis for rewards: we already discussed gaming of contribution metrics previously if you only rely on metrics for rewards in the community. In addition, if people perceive that rewards and recognition are mostly based on the volume of work (or input), you run the risk of discouraging people who aren’t able to devote as much time to the project or people who are getting started in the community. People do not join open source communities just to do more work and we do not want to lose sight of their intrinsic motivation.
-
Don’t present metrics without a proper context: even when you get asked what sounds like a straightforward question like "how many contributors do you have in your community?", it is always helpful to get some context behind the question. Depending on who is asking the question, they’re usually asking for something slightly different. For example, the total number of contributors in project’s history may be appropriate in one context, but in another the growth of contributors over a time period may be what the questioner is really after. Also what do they mean by contributors? Do they also want to include people contributing to internationalization, issue triage, event planning, etc.? So before we simply point people to a set of metrics, it is helpful to understand the context or even motivation behind their question.
-
Ignoring non-metrics: As discussed previously, not everything is easily measurable or quantifiable. Even if we have a well defined and polished metrics dashboard, it should not stop us from continuing to have human conversation with community members to keep a pulse on what is happening in the community and encourage community members to point us to what we are not able to see in our metrics.