-
Notifications
You must be signed in to change notification settings - Fork 0
GSOD 2021 Case Study
Organization or Project: The R Project (Season of Docs 2021 wiki)
Organization Description: The R Project maintains and develops R, an integrated suite of software facilities for data manipulation, calculation, and graphical display. There are thousands of add-on packages available through the Comprehensive R Archive Network (CRAN), extending the core functionality of R to cover a wide range of tools for modern data science.
Authors: Matt Bannert (@mbannert), Noa Tamir (@noatamir), Heather Turner (@hturner)
Date: 30 November 2021
What Problem Were You Trying to Solve With New or Improved Documentation?
useR!, the annual conference of the R Project, is organized by a different team of community organizers each year. Before this Season of Docs project, information was mainly passed from one organizing team to the next in the form of a large repository of files produced during the organization. This was hard to navigate, with a lot of extraneous local or event-specific detail. Long-term history was not well-preserved, with some pieces of information gathered centrally, but other information scattered in direct messages, emails, or documents that were not passed on, or stored only in people's memories. This meant that organizers often re-evaluated/re-designed similar tools or processes and found it easier to ask questions of past organizers than to find out information for themselves. It also made it hard to ensure that improvements made one year were carried over to the next.
New and improved documentation was required to ease the burden on both conference organizers and the voluntary groups that support them across the years, such as the R Foundation Conference Committee. This became even more important with the higher demands on organizers due to the need for a strong online component.
A brief summary of the original organization proposal
The proposal for Season of Docs was to gather facts about previous useR! conferences along with existing pieces of documentation and organize this information into two main outputs: a useR! knowledgebase and an interactive information board. The knowledgebase was proposed to take the form of an online book, inspired by examples such as the satRdays knowledgebase and the DISCOVER cookbook. The information board was proposed as a dashboard to interactively browse historical information.
Together, the knowledgebase and information board would provide comprehensive, centralized documentation that would help to sustain a high-quality conference and to facilitate longer-term planning.
How did you come up with your Season of Docs proposal? What process did your organization use to decide on an idea? How did you solicit and incorporate feedback?
The 2021 edition of useR! was explicitly advertised as an online conference early on. Doing so allowed for an unprecedented inclusivity, welcoming participants from more than 130 countries. This global community effort also provided the seeding ground for this Season of Docs project.
@benubah asked on the useR! 2021 Slack if the R Project was taking part in Season of Docs and came up with the original idea for a useR! information board. He discussed it by DM with @hturner, who suggested the knowledgebase idea.
@hturner teamed up with another useR! 2021 organizer, @mbannert, to take on the role of administrators, setting up a GitHub organization based on the R Project's Summer of Code GitHub. @hturner drafted proposals based on each of the project ideas discussed on Slack. These ideas were shared for feedback with the R Foundation, the R Project's Google Summer of Code Google Group (gsoc-r) and R community Slack groups.
Several volunteers stepped forward to help with the project, forming a steering committee. In consultation with this committee it was decided to merge the ideas for the final proposal.
Include a short section on your budget. How did you estimate the work? Were there any unexpected expenses? Did you end up spending less than the grant award? Did you have other funds outside of Season of Docs that you were able to use?
There was experience in the team of working as freelancers and hiring freelancers, both in high-income countries (in Europe) and lower-middle-income countries (in Africa and Asia). This helped us to set appropriate rates for budgeting.
Given the two components of the project and to maximise the opportunities for candidates, we budgeted for two technical writers from lower-middle-income countries to work approximately 30 hours/month over 6 months and one senior writer from a high-income country to work 4-6 hours/month over 6 months. Members of the steering committee were given the option of a $500 stipend.
We did not have any additional funding, so we also needed to budget for payment processing fees. The amount was somewhat uncertain in advance, but the 3% that we allowed covered these costs.
Who worked on this project (use usernames if requested by participants)? How did you find and hire your technical writer? How did you find other volunteers or paid participants? What roles did they have? Did anyone drop out? What did you learn about recruiting, communication, and project management?
The people working on this project were:
- Noa Tamir (@noatamir), senior writer/project lead.
- Ben Ubah (@benubah), technical writer (information board).
- Sangeet Moy Das (@sangeetm), technical writer (knowledgebase), May - August.
- Andrea Sánchez-Tapia (@andreasancheztapia), technical writer (knowledgebase), October - December.
The following people contributed as volunteers:
- Yanina Bellini Saibene (@yabellini), Nathalie Vialaneix (@tuxette) (writer, steering committee)
- John Nash (@nashjc) (copyeditor, steering committee)
- Heather Turner (@hturner), Matt Bannert (@mbannert) (Season of Docs admins, steering committee)
- Achim Zeleis (@zeileis) (steering committee)
Expressions of interest from technical writers were invited when emailing the gsoc-r mailing list and posting on the R community Slack groups. Potential writers were asked to email the admins with links to their technical writing work or portfolio/résumé/CV. We received a few applications in response and these candidates helped to finalise the proposal.
After the R Project proposal was accepted for Season of Docs, we received further expressions of interest, which we continued to accept until 2 weeks before the technical writer hiring deadline. However, in the end we selected two candidates from those that expressed an early interest in our project. In addition to the technical skills shared by other candidates, these candidates had a background in and a commitment to the R community, as well as some knowledge of the useR! conference that would be beneficial for the project.
Alongside the two technical writers, we directly recruited @noatamir as senior writer, to help supervise the project. This selection was based on her contribution to existing documentation about useR!, as well as relevant managerial experience.
The steering committee was formed from the volunteers that stepped forward when we shared the proposal for feedback. Between them, they had a wealth of experience covering useR! organization, Google Summer of Code mentoring and technical writing.
Unfortunately, @sangeetm had to withdraw from the project in September for personal reasons. We were able to recruit @andreasancheztapia from the useR! 2021 organizing team as a replacement - given her experience she was able to get up to speed very quickly.
We learnt it was important to keep our Season of Docs wiki up-to-date so that potential candidates for paid roles were aware of the current status of the recruitment process. Having the steering committee helped to identify potential candidates when we need to quickly find a replacement writer, since we could rely on their network.
From the start, we set a regular pattern of "sync" meetings. The technical writers met with the senior writer weekly/bi-weekly, then all the writers met with the steering committee mid-way and at the end of two project phases. This helped to monitor progress and pick up on issues in the team.
Give a short overview of the timeline of your project (indicate estimated end date or intermediate milestones if project is ongoing)
After our project was accepted, we focused on getting our Open Collective account set up. We investigated the possibility of the R Foundation acting as fiscal host, but this was not feasible, so we applied to be hosted by the Open Source Collective. This took a few weeks and we held back on hiring technical writers until we could confirm that we would be able to receive the funding.
With the Open Collective set up, we selected the technical writers and informed both successful and unsuccessful candidates. We invited the writers and steering committee members to join a channel on the useR! 2021 Slack, where we began to get to know each other. The plan for the remainder of the project was as follows:
| Stage | Knowledgebase | Information board | Completed By |
|---|---|---|---|
| Kick-off meeting | 17 May | ||
| Phase 1a | Deploy book with skeleton sections | Deploy prototype with initial figures based on existing data | 2 July |
| Mid-phase 1 meeting | 2 July | ||
| Phase 1b | Gather information; fill out initial sections | Add summaries based on existing/easily accessible data | 30 July |
| Delivery Phase 1/Kick-off Phase 2 meeting | 30 July | ||
| Phase 2a | Fill out further sections based on gathered information | Gather information for further summaries | 12 October |
| Mid-phase 2 meeting | 12 October | ||
| Phase 2b | Complete sections of high-priority for useR! 2022 | Complete summaries prioritised by steering committee | 16 November |
| Delivery Phase 2 meeting | 16 November | ||
| Prepare and submit case study | 30 November |
For both the knowledgebase and the information board, the regular meetings provided the opportunity to set more specific goals for the next half-phase, in collaboration with the steering committee. The work plan was adapted to make the most of the time we had budgeted for.
What was created, updated, or otherwise changed? Include links to published documentation if available. Were there any deliverables in the proposal that did not get created? List those as well.
We created a knowledgebase collecting, streamlining and updating information from different sources. Information about previous years was mostly gathered from different git repositories and through exchange with the R Foundation Conference Committee.
The second output is an information board with statistics about the useR! conferences.
Both the information board and knowledgebase make use of R language based static website generators to create and update content. The knowledgebase makes use of the R {bookdown} package to render an online book from markdown. The information board is rendered using the {flexdashboard} R package. We make use of continuous integration / continuous development techniques to facilitate deployment to GitLab Pages. Neither documentation output existed before the Season of Docs project.
What metrics did you choose to measure the success of the project? Were you able to collect those metrics? Did the metrics correlate well or poorly with the outcomes you wanted for the project? Did your metrics change since your proposal?
The metrics we proposed and the current status is outlined below:
-
At least 80% of the conference-related markdown files previously maintained in the useRorganization GitLab repository and the Forwards conferences and event_best_practices GitHub repositories are deprecated by incorporating and updating that information in the knowledgebase.
At least 80% of the relevant content from the useRorganization GitHub and 100% of the relevant content from the Forwards repositories has been incorporated.
-
The information board documents key information (e.g. keynote speakers, program committee members, presentation numbers) for at least 5 years, up to the present year.
The information board documents key information on program (contributed presentations, tutorials, keynotes, program committee members), organisation (sponsors, key dates, registration fees, organization committee members), participants (registration types, gender, ethnicity, country of residence) and conference location for at least 5 years, up to the present year.
-
The new dashboard is used by useR! organizers and the wider R community. This would be measured by searching for links to the dashboard in the organizer communication (e.g. Slack workspace) and internet (e.g. with Google Search Console).
The R Foundation Conference Committee and the R Foundation funded conference assistant have been referring to the knowledgebase and information board on the organizer Slack and in email correspondence. This has provided information on previous numbers of attendees and keynote speakers (information board) as well as how tutorials are organized and information on captioning services (knowledgebase). We set up analytics on both sites in September and the following numbers of unique visitors were recorded to date:
Country Knowledgebase Information board USA 82 12 Argentina 12 9 Germany 11 14 UK 8 18 Switzerland 4 11 Nigeria 3 16 France 2 2 Canada 1 0 All of these countries are represented on the project team/steering committee and it seems likely that many of the "unique" visitors are due to visiting on different devices or from different venues. However, useR! 2022 is being organized by a team based in the USA, so it seems likely that the high count of 82 visitors to the knowledgebase reflects at least some use by the new useR! organizers, who have not been directly involved in this project.
We did not verify the sites on Google Search Console yet as we have not publicized them widely, this could be considered in future.
-
At least two community members that are not directly involved in the project contribute corrections/updates to the documentation or data sources it relies on, by making a pull request or commit to the git repository.
There were no pull requests made by contributors outside the project. However, several members of the useR! 2021 team not directly involved in the project contributed content via issues, including
The metrics that measure the scope of the documentation demonstrate the success of the project in incorporating and extending existing information. In future, we would not expect the scope to expand too much, but will require useR! organizers and other community members to help keep the documentation up-to-date. For monitoring contribution, it seems it is important to track issues, not just commits/pull requests.
To be effective, the documentation should be used by current and incoming organizers. Using the current metrics, we are not be able to track such use directly. So, we will provide different organizing teams with short links to help identify when visitors are on a particular organizing team. Similar links could help to track visits from other sources. We expect both the knowledgebase and information board to be used more widely over the coming months, now that the documentation is more comprehensive.
Qualitative feedback from the organizing team could also be collected by the R Foundation Conference Committee or the recently assembled global useR! working group, to help assess the usefulness of the documentation.
What went well? What was unexpected? What hurdles or setbacks did you face? Do you consider your project successful? Why or why not? (If it's too early to tell, explain when you expect to be able to judge the success of your project.)
The project management worked well and helped us to overcome the hurdles we faced. We were able to integrate and coordinate a diverse group of people across wide-ranging time zones. We would definitely organize the team in a similar way for future projects -- in particular hiring someone for the project lead role really paid off, it would have been difficult for the admins or other volunteers to manage the project on top of other work.
In August multiple contributors were unavailable for a several weeks due to various personal reasons. As mentioned earlier, one of our technical writers had to withdraw after a period of absence, so the team had to recruit a replacement and regroup. To make up for the lost time, we had to increase our pace in the last two months of the project and the writers continued to work a little beyond the planned Nov 16 delivery date to complete some issues. An additional, but minor hurdle was the availability of historical information to the entire team.
Given the size and scope of the useR! conference, useR! 2022 was already in the making when the Season of Docs project started. Hence we expect our work to graze the upcoming useR! but have much larger effect in future years, when we hope that teams will get used to referring to the documentation from the start. The information board can also be useful at an even earlier stage, when a prospective team is preparing a bid to host a useR! conference.
So, we consider the project successful in terms of delivering a first draft of the knowledgebase and information board and in terms of being a resource for volunteers with long-term involvement with useR! as they support the current organizers. But we will need to continue monitoring use of the documentation up to next useR! (June, 2022) and beyond, to determine whether useR! organizers refer to it of their own accord and help to update it based on their own experience.
In 2-4 paragraphs, summarize your project experience. What did you learn? What would you do differently in the future? What advice would you give projects trying to solve a similar problem with documentation?
The overall project experience was very positive. First, we are confident to have started a resource with value to our community. Second, it was great to see a heterogeneous, global team successfully conduct the project and deliver on time. Third, we were able to overcome challenges thanks to organization, team effort and the apt duration of Google Season of Docs.
In future projects we would do an assessment of key risks such as team members dropping out, or lack of momentum over August when many contributors take a long vacation. We would plan responses to cover the most common risks from the start. Doing so reduces insecurity, eases the burden of the remaining team and supports team members by adapting to their circumstances.
Our team consisted of a project lead, two main contributors, a group of supporting volunteers and two Season of Docs admins. We learned how to meet each person at their own pace throughout different phases of their contribution and stages of the project. The project lead role helped us manage different energy levels and preferred working hours, while organizing and leading progress and content meetings along the way. Based on this experience we advise other projects with multiple contributors to carefully choose a suitable person for project lead responsibilities.
We would like to acknowledge authors of documentation incorporated into the new knowledgebase who are not mentioned above. In particular, Liz Hare, for her documentation on good accessibility practices.
Project links
- Original Proposal: https://github.com/rstats-gsod/gsod2021/wiki/GSOD-2021-Proposal
- Meeting minutes: https://gitlab.com/rconf/userorganization/-/wikis/GSOD-Meeting-Minutes-TOC
- Information board Gitlab: https://gitlab.com/rconf/userinfoboard
- Knowledgebase Gitlab: https://gitlab.com/rconf/userknowledgebase