Organization or Project: CHAOSS
Organization Description: CHAOSS is a Linux Foundation project focused on creating metrics, metrics models, and software to better understand open source community health on a global scale. CHAOSS is an acronym for Community Health Analytics in Open Source Software. Open source software is critically important for both individuals and organizations. This importance raises questions about how we understand the health of the open-source projects we rely on. Unhealthy projects can have negative impacts on the community involved in the project as well as organizations that rely on such projects.
Author of this Case Study: Elizabeth Barron
CHAOSS is not a large project, but it has a lot of moving parts. It is not your typical software-only kind of open source project. Guiding newcomers on where to go, and providing helpful information for them to get started has always been challenging for us. With the Google Season of Docs project we were hoping to make this newcomer journey smoother through documentation.
We had several ideas for improving documentation across the entire project, but these are the two sub-projects that our technical writers worked on.
Work with CHAOSS members to develop a community handbook. Currently, working groups have developed their own ways of working and documented their disparate processes to varying degrees. The goal of the community handbook is to centralize critical information and standardize parts of it across the CHAOSS project. The handbook should serve as an instruction manual for existing and new community members on how to get work done in the CHAOSS project. This project involves a creative component of collecting and organizing content for the handbook as well as a technical component of defining how to represent the handbook.
Knowledge/Skills Involved: Being a good human with good ethics. GitHub-Flavored Markdown and Workflow, GitHub Pages, HTML Inlines/Tables, Vector/Raster Images Requirements.
Aims of the project: Create a Community Handbook for the CHAOSS project.
Mentors: Elizabeth Barron
References: https://github.com/chaoss/governance/tree/master/community-handbook
License: MIT
Work with CHAOSS members, including Augur developers, to document how we move from exploration of Augur's schema with Jupyter notebooks to the development of new Augur Endpoints, and new CHAOSS Metrics. Currently, the iterative process of metric definition and implementation occurs in an ad hoc manner. The goal of the community Augur developer guide is to make the process of asking a question using Python or SQL, into formalized CHAOSS Metrics, and implemented Augur code. The guide should serve as an instruction manual for existing and new community members on how to answer the questions they have using Augur data, and then helping contribute to the Augur software and the CHAOSS project. This project involves a creative component of collecting and organizing content for the handbook as well as a technical component of defining how to represent the handbook.
Knowledge/Skills Involved: Being a good human with good ethics. Familiarity with RestructuredText, GitHub Pages, and image placement.
Aims of the project: Create an Exploration to Metrics Guide for the Augur and CHAOSS projects.
Mentors: @sgoggins, @ccarterlandis, @gabe-heim
Sean Goggins, Carter Landis, Gabe Heim
References: https://oss-augur.readthedocs.io/en/master/
License: MIT
We had several ideas covering various areas of CHAOSS, because there is always room for improvement in documentation! We solicited ideas from the various stakeholders in our community openly in GitHub and then narrowed those ideas down to four sub-projects that would address different aspects of the project. Based on the interest from potential mentees, we ended up with two. All ideas were centered around helping newcomers find their way and increasing participation from potential contributors.
Based on the approximate number of hours agreed upon by the mentors and contributors, we agreed on a budget of $5000 USD per writer, for a total of $15,000 for the project.
- Ruth Ikegah
- Xiaoya Xia
- Satyam Kumar
All participants remained on task and on budget. All participants regularly checked in with their mentors, and all participants successfully completed their parts. We couldn’t be happier with their progress and their collaboration.
At the beginning of the project, each participant opened a string of issues that would address the various tasks and milestones across the length of the project. Each issue correlated with individual timelines. Some of the key milestones were:
- Perform an audit of current documentation
- Restructure document organization to improve the flow of information
- Identify which documents needed to be updated or changed
- Idenfity which documents needed to be created from scratch
- Upload documents to the website’s new Knowledge Base
This project was also intermingled with a Google Summer of Code project which implemented a new Knowledge Base for the CHAOSS website, and a complete website redesign. Pieces of the final milestone for this project were dependent on those other projects, and are being finalized this week.
What was created, updated, or otherwise changed? Include links to published documentation if available. Were there any deliverables in the proposal that did not get created? List those as well. Did this project result in any new or updated processes or procedures in your organization?
This project resulted in a complete overhaul of the current collection of documents we had compiled around community policies and procedures, information about how work gets done in the project, ways to contribute, and other similar topics. The Community Handbook documentation now lies in a fully searchable and indexed Wordpress Community Knowledgebase, and the Augur documentation lies here.
Additionally, because this project highlighted the need for periodic updating and reviewing of documentation, a few of our contributors will be working on a bot that submits an open issue as a reminder to review current docs. This will be a new procedure for us and will hopefully help keep documentation top of mind for us and offer a way for newcomers to help review our documents for accuracy, readability, and usefulness.
What metrics did you choose to measure the success of the project? Were you able to collect those metrics? Did the metrics correlate well or poorly with the behaviors or outcomes you wanted for the project? Did your metrics change since your proposal? Did you add or remove any metrics? How often do you intend to collect metrics going forward?
We measure the community response to documentation through a community survey we run periodically. We ran the survey before the documentation overhaul was complete, and we will run it again in a few months to determine the efficacy of the changes made to the documentation. We ask questions around documentation accessibility, usability, and discoverability. Of particular interest is how our newcomers rate our documentation in these areas.
What went well? What was unexpected? What hurdles or setbacks did you face? Do you consider your project successful? Why or why not? (If it's too early to tell, explain when you expect to be able to judge the success of your project.)
Because this project was tied to a few other projects involving an entire website overhaul, we relied on a lot of cross-team collaboration and coordination. Some of the timelines of the other projects, as well as limitations of personal schedules did have an effect on implementation of a few final pieces of the documentation project. But overall, given the number of community members who were involved, and the other moving parts, we are extremely proud of the way our writers were able to collaborate and adjust when needed.
We will be able to gauge success of our documentation upgrades after a few months of community engagement. As mentioned before, we plan to run our community survey again.
In 2-4 paragraphs, summarize your project experience. Highlight what you learned, and what you would choose to do differently in the future. What advice would you give to other projects trying to solve a similar problem with documentation?
It is difficult for newcomers to a project to jump right into improving documentation as their first contribution, although in open source, we often use this as a path for new contributors. It’s hard for someone to offer meaningful contributions until they fully understand the context of the project and the moving parts that go into it, so it seems counterintuitive that this is often the entry point. It’s for this reason we wanted our documentation to be more accessible, discoverable, and useful particularly for newcomers, so that this point of entry makes more sense, and the experience of making a first contribution is smoother.
Our advice would be to highly engage documentation writers on every level of the community, whether they are writing technical documentation or not. Oftentimes with time-sensitive projects like GSoD, there is a temptation to jump right in to changing documents and making PRs. However, offering space, time, and opportunities for new writers to be a part of the community through other channels like community meetings, Slack discussions, or other contribution pathways allows for a better documentation product and end result. Having the full context of how the project works and what the policies and procedures are that are actually in place means that the accuracy of documentation will be greater, and it will require less oversight from seasoned contributors.
Additionally, although it’s risky to intermingle different time-based project initiatives, and requires a lot of patience and collaboration from numerous teams, it can be a highly valuable experience for the writers, as it mimics real-world situations where things don’t often occur in siloed bubbles.
And finally, setting up a process for continuing the work started by writers is crucial, and a mistake we made in previous years of working with the GSoD program. Although you hope that your interns and writers will remain a part of your community, there is no guarantee. So including a process for how documentation will continue to be improved and iterated upon is crucial to the long-term success of the project and the GSoD work.