Raw data from the Understanding Challenges survey along with a direct…

…ory for people to use for draft interpretations

Signed-off-by: Dawn M. Foster <dawn@dawnfoster.com>

challenges_survey/README.md

@@ -0,0 +1,14 @@
+# Understanding Challenges Survey
+
+In August and September, the CHAOSS project ran a survey of existing and past users of CHAOSS tools and metrics designed to help us better understand the barriers and challenges that make it difficult for people to gain meaningful, empirically-driven community health insights using CHAOSS tools and metrics. The [blog post](https://chaoss.community/survey-help-the-chaoss-project-improve-our-tools-and-metrics) has more details about the survey.
+
+In the spirit of open source, we are making the raw data available to the CHAOSS data science community to encourage people to work together to interpret the data. The raw data with personally identifiable data redacted can be found in the raw_data directory in this repo. The CSV file contains everything except the free form responses. Because there were very few reponses from certain groups (e.g., universities, government, nonprofit), attaching the free form responses to rest of the data made it possible to identify the person / group responsible for certain comments, so the order of the free form comments in the text documents has had the order randomized and some text redacted to prevent identification. Here are the filenames with the raw data:
+* Clean_CHAOSS_Understanding_Challenges_Survey.csv - responses to the quantitative parts of the survey
+* other_challenges_freeform_redacted.txt - free form text responses to the question, "What other challenges have you faced that weren’t in the above list or what else would you like to see us improve?"
+* strengths_freeform_redacted.txt - free form text responses to the question, "What do you see as CHAOSS project strengths (what do you love about CHAOSS)?"
+* understanding_challenges_graphs.pdf - simple graphs produced by Google forms displaying the responses for each question
+
+We've created a draft_interpretations directory where community members can upload your interpretations of the data using a Pull Request to create files in that directory.
+
+We will also create an "official" or final report that will be shared with the rest of the CHAOSS community along with recommendations for improvements. If you are looking for those results, you'll be able to find them here in this repository in October or November.
+

challenges_survey/draft_interpretations/README.md

@@ -0,0 +1,3 @@
+# Draft Interpretations
+
+If you would like to share your interpretation of the survey data, please create a PR and place your files in this directory. If you have multiple files, please create a subdirectory.

challenges_survey/raw_data/Clean_CHAOSS_Understanding_Challenges_Survey.csv

@@ -0,0 +1,46 @@
+What type of organization do you work for?,Do you work in an Open Source Program Office (OSPO) or similar open source team?,Which of these best describes your role or position?,"Have you contributed to the CHAOSS project (including non-code contributions, e.g., metrics definitions, documentation, presentations, blog posts, meeting attendance)? ",Which of these have you used or tried to use?,How long have you been using CHAOSS tools or custom code that implements CHAOSS metrics?,"Rank order the challenges you have faced using CHAOSS tools. If any don’t apply 
+to you, select the option labeled ‘NA’ (Not Applicable).
+1 is most challenging and 7 is least challenging.  [Installing / configuring software]","Rank order the challenges you have faced using CHAOSS tools. If any don’t apply 
+to you, select the option labeled ‘NA’ (Not Applicable).
+1 is most challenging and 7 is least challenging.  [Maintaining software over time]","Rank order the challenges you have faced using CHAOSS tools. If any don’t apply 
+to you, select the option labeled ‘NA’ (Not Applicable).
+1 is most challenging and 7 is least challenging.  [Cleaning up the data (e.g., merge duplicate contributors, company affiliation)]","Rank order the challenges you have faced using CHAOSS tools. If any don’t apply 
+to you, select the option labeled ‘NA’ (Not Applicable).
+1 is most challenging and 7 is least challenging.  [Finding the data / metrics you want to use]","Rank order the challenges you have faced using CHAOSS tools. If any don’t apply 
+to you, select the option labeled ‘NA’ (Not Applicable).
+1 is most challenging and 7 is least challenging.  [Drawing meaningful insights out of the data]","Rank order the challenges you have faced using CHAOSS tools. If any don’t apply 
+to you, select the option labeled ‘NA’ (Not Applicable).
+1 is most challenging and 7 is least challenging.  [Communicating meaningful insights to others, including executives]","Rank order the challenges you have faced using CHAOSS tools. If any don’t apply 
+to you, select the option labeled ‘NA’ (Not Applicable).
+1 is most challenging and 7 is least challenging.  [Getting others within your company / community to use the software]"
+University or other academic institution,Yes,"Development or operations focused (e.g., developer, sys admin)",Currently contributing to the CHAOSS project,Augur,Less than 1 year,3,2,4,1,NA,NA,NA
+For-profit company,Yes,"Leadership (e.g., primarily manage other people)","Past contributor, but no longer contributing",GrimoireLab (including users of Bitergia’s platform and Cauldron),More than 3 years,1,2,3,7,4,6,2
+None of the above,No,consultancy,"Past contributor, but no longer contributing",GrimoireLab (including users of Bitergia’s platform and Cauldron),More than 3 years,5,NA,7,3,2,1,4
+University or other academic institution,Yes,"Leadership (e.g., primarily manage other people)",Currently contributing to the CHAOSS project,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",2 - 3 years,1,2,3,5,4,NA,NA
+For-profit company,Yes,"Development or operations focused (e.g., developer, sys admin)",Currently contributing to the CHAOSS project,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",1 - 2 years,6,5,1,4,3,7,NA
+For-profit company,Yes,"Leadership (e.g., primarily manage other people)",Currently contributing to the CHAOSS project,Augur,1 - 2 years,2,1,1,1,1,3,4
+University or other academic institution,No,"Community focused (e.g., community manager)",Currently contributing to the CHAOSS project,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",Less than 1 year,1,NA,4,2,7,7,NA
+None of the above,No,"Development or operations focused (e.g., developer, sys admin)",Never contributed to CHAOSS,GrimoireLab (including users of Bitergia’s platform and Cauldron),I haven’t used CHAOSS tools,NA,NA,4,4,5,NA,NA
+For-profit company,Yes,"Leadership (e.g., primarily manage other people)",Currently contributing to the CHAOSS project,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",More than 3 years,2,6,5,7,7,7,5
+For-profit company,Yes,"Community focused (e.g., community manager)","Past contributor, but no longer contributing",GrimoireLab (including users of Bitergia’s platform and Cauldron),2 - 3 years,1,NA,2,1,2,2,2
+None of the above,No,Consultant,"Past contributor, but no longer contributing","Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",More than 3 years,1,7,6,5,3,2,4
+For-profit company,Yes,"OSPO Lead, direct activities and liaison with CISO/Eng/Legal",Never contributed to CHAOSS,Commercial tools with project Health metrics like Snyk,I haven’t used CHAOSS tools,3,2,NA,NA,NA,NA,2
+For-profit company,No,"Community focused (e.g., community manager)",Currently contributing to the CHAOSS project,GrimoireLab (including users of Bitergia’s platform and Cauldron),2 - 3 years,NA,NA,1,3,1,3,4
+For-profit company,Yes,"Development or operations focused (e.g., developer, sys admin)","Past contributor, but no longer contributing","Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",1 - 2 years,5,NA,NA,2,1,1,1
+For-profit company,Yes,Program Manager,Currently contributing to the CHAOSS project,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron), DEI.md",1 - 2 years,3,NA,NA,3,3,3,NA
+Nonprofit,Yes,"Community focused (e.g., community manager)",Currently contributing to the CHAOSS project,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",Less than 1 year,NA,NA,NA,6,6,6,2
+None of the above,Yes,"Development or operations focused (e.g., developer, sys admin)",Never contributed to CHAOSS,GrimoireLab (including users of Bitergia’s platform and Cauldron),More than 3 years,6,3,1,4,2,1,7
+For-profit company,Yes,"Community focused (e.g., community manager)",Currently contributing to the CHAOSS project,"MergeStat, CNCF DevStats",I haven’t used CHAOSS tools,4,6,5,1,2,3,7
+For-profit company,Yes,"Data focused (e.g., data analysis, data science)",Currently contributing to the CHAOSS project,GrimoireLab (including users of Bitergia’s platform and Cauldron),2 - 3 years,7,7,7,7,7,7,3
+Government,Yes,"Community focused (e.g., community manager)",Never contributed to CHAOSS,"Don’t know, not sure, or haven’t used any tools",I haven’t used CHAOSS tools,NA,NA,NA,NA,NA,NA,NA
+Nonprofit,Yes,"Community focused (e.g., community manager)",Never contributed to CHAOSS,Augur,Less than 1 year,NA,NA,NA,3,3,NA,NA
+For-profit company,Yes,the first three,Currently contributing to the CHAOSS project,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",More than 3 years,7,7,7,7,3,3,7
+For-profit company,Yes,"Data focused (e.g., data analysis, data science)",Currently contributing to the CHAOSS project,GrimoireLab (including users of Bitergia’s platform and Cauldron),2 - 3 years,NA,NA,NA,2,2,2,1
+For-profit company,No,"Data focused (e.g., data analysis, data science)",Currently contributing to the CHAOSS project,GrimoireLab (including users of Bitergia’s platform and Cauldron),2 - 3 years,3,6,2,5,6,2,1
+For-profit company,Yes,"Community focused (e.g., community manager)",Never contributed to CHAOSS,"Don’t know, not sure, or haven’t used any tools",I haven’t used CHAOSS tools,2,3,1,1,1,1,2
+For-profit company,No,"Community focused (e.g., community manager)",Never contributed to CHAOSS,GrimoireLab (including users of Bitergia’s platform and Cauldron),More than 3 years,1,4,3,3,6,6,2
+For-profit company,Yes,"Community focused (e.g., community manager)",Currently contributing to the CHAOSS project,"Don’t know, not sure, or haven’t used any tools",I haven’t used CHAOSS tools,7,NA,NA,NA,NA,NA,7
+For-profit company,Yes,"Leadership (e.g., primarily manage other people)",Currently contributing to the CHAOSS project,Augur,1 - 2 years,7,7,5,7,6,7,6
+For-profit company,Yes,"Leadership (e.g., primarily manage other people)",Never contributed to CHAOSS,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",Less than 1 year,2,3,6,4,4,6,4
+For-profit company,Yes,"Leadership (e.g., primarily manage other people)",Never contributed to CHAOSS,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",1 - 2 years,5,5,5,5,5,5,5
+None of the above,No,"Data focused (e.g., data analysis, data science)",Currently contributing to the CHAOSS project,"Augur, GrimoireLab (including users of Bitergia’s platform and Cauldron)",More than 3 years,1,2,3,6,5,7,4

challenges_survey/raw_data/other_challenges_freeform_redacted.txt

@@ -0,0 +1,47 @@
+What other challenges have you faced that weren’t in the above list or what else would you like to see us improve?
+
+Getting either piece of software running locally proved impossible for me :)   It's hard to understand what is "official" CHAOSS software (like what is the relationship to compass?) or how things move from the metrics/models into the software. Understanding the relationship between the software and the metrics can be difficult, things aren't always named the same and the methods of calculation are not always transparent in the software so you can't tell if it is really doing what you think it is.   I would really like some lighter weight ways to play with the metrics. I was excited by the prospect of Jupyter notebooks but then it turned out those also rely on Augur which could be fine but then some better instruction on getting just the Augur db running would be good. A simplified Augur install that is just the bare minimum for working with a data snapshot would be nice. Multiple instances of documentation also make working with Augur hard. There also Augur meetings on the calendar but I'm not sure if they are actually happening, and I'm not sure if they are just for people doing development work on Augur, as opposed to users.   I worked with Cauldron.io and thought it was a great tool but could not get GL running locally so my experimenting was limited to what was offered there. As a [REDACTED] I don't really have a budget to pay for metrics as a service just to experiment or to try and get other people involved, so I greatly appreciated the cauldron service.    The focus on OSPOs was a surprise to me as I got more involved in the project, it's understandable but makes it more difficult for people not in an OSPO to be seen as an audience for metrics/software, which is unfortunate because I think there is still a lot of value in CHAOSS metrics for people who are not part of an official OSPO.
+
+More clarity/documentation in data model (for comparison against other data sources); OpenSearch backend is incompatible with other internal tooling so our implementation is a bit of an island :/ 
+
+Contributing has itself been a challenge, since documentation is woefully incomplete and (at times) inaccurate
+
+From an OSPO perspective, I feel that OpenSSF Scorecard and Open Source Review Toolkit are quite valuable tools and resources ("easy" to set up and get a report). It would be great to see CHAOSS tools at the same level of maturity and value, or even integration with such tools, to get more adoption in the industry. On the other hand, the diversity of tools and setups related to open source development and contribution makes it so hard to find "the right" tools to get valuable information about people, activity, and "performance".
+
+Any new metric should have use cases associated where the usefulness of the metric becomes evident
+
+Both GrimoireLab and Augur don't really provide a UX that is fit for a product. Rather they are fairly complicated back-ends that require a significant amount of configuration to set up. For a well funded OSPO that has a clear list of the data sources they want to track, this may be useful, but for something like [REDACTED], this means that many of our needs are not met.  In our case as [REDACTED], we need to provide an experience for [REDACTED] that encourages them to track the community health of projects they use and participate in. This requires a certain kind of UX as well as better integration with other data sources and services that we provide. Particularly with GrimoireLab we found that building on top of the current architecture was fairly difficult. 
+
+grimoire sigils should support opensearch / pivot away from kibiter asap
+
+Overwhelming data.
+
+Integrating grimorelab with custom dashboards and frontend/backend software seemed tricky as far as ive heard from the team that was responsible for that. I dont know the details though, sounded like there was some architectural tech debt (not sure what that means) that made it hard to for the developers to integrate.
+
+Bitergia tools are quite complicated and take a lot of time to set up properly.
+
+self hosting is becoming really difficult, esp for remote only orgs
+
+This was more of a product issue (when I used Bitergia), but a few times we had infrastructure issues so that an up-to-date data wasn't available for an extended period. For corporate users (especially in technology companies), the main reason why they go with an outside vendor is so that they don't need to manage infrastructure/software.
+
+No CHAOSS tools make it easy to compare a large number of repos.  Many CHAOSS metrics aren't quantitative, so evaluating them requires manual examination. CHAOSS metrics don't cover marketing metrics like social media mentions.
+
+Documentation seems really good but there are a LOT of steps - I have such a limited amount of time that by the time I read through the documents to remember where I left off last time, I've already run out of time to do any experimentation.
+
+The most challenging part is to communicate them to C level.
+
+Estimating how much time and effort using tools would take, compared to ad hoc methods of assessing similar questions.
+
+ability to see and understand what other people are doing in real situation so I can leverage that work, and compare with my own
+
+Issues and PR backlog 
+
+Measuring the health and success of community (marketing) activities
+
+Overall I find it complex to understand what CHAOSS is actually about and how the tools relate to what is described. They all seem quite complex and difficult to get started or require very special data analysis expertise. Maybe adding pre-requirements to usem them?
+
+Docker compose
+
+Creating production environments (security, set up backups, durability), updating, migrating to newer components (OpenSearch).
+
+Grimoirelab did not work well for the size of our OSPO. We find Augur easy to install and use. 

challenges_survey/raw_data/strengths_freeform_redacted.txt

@@ -0,0 +1,62 @@
+What do you see as CHAOSS project strengths (what do you love about CHAOSS)?
+
+The depth of expertise in the community
+
+CHAOSS is exceptional at community health. The community generating community health metrics is unusually healthy. 
+It is a model for good practices.
+
+Being committed to open-source is becoming frustratingly less common, so I'm glad to see an organization insisting that their work will belong to the people, and not to any hypothetical corporate backers.
+
+The passion and commitment of the CHAOSS community to provide a set of valuable metrics to describe an open source projects.
+
+Agreements over definitions
+
+CHAOSS has an active and welcoming community. The projects are interesting and useful.
+
+ease of community involvement and insight into metrics
+
+Standard metrics, and good tools
+
+The project is open and welcoming and is doing really important work! I really appreciate the extent to which newbies are able to participate.
+
+Seems interesting. Havent interacted with them much though, but it seems like they have a good mission/goal. If CHAOSS is at all related to OSI, maybe this thing a coworker sent me recently could be applicable? https://yakshav.es/non-thoughts-on-the-osi/
+
+- Creating a welcoming space for open source project leaders to connect and discuss community health indicators.  - Educating community members on the existing CHAOSS metrics and metrics models and empowering them to develop new ones to address their needs.
+
+The existence of a unified, standardized set of metrics is invaluable for determining community health and value.
+
+Great mission, great educational outreach, great conference presentations, genuinely nice people.
+
+open source, community based, non-commercial 
+
+Well respected/regarded in open source communities. Good community culture. 
+
+The promise of getting metadata about the health of my open source.
+
+Great community, solid ideas and great tooling available
+
+The community!
+
+News Ideas, and family like. 
+
+Only place I know of where these OS community metrics exist.
+
+great community, lots of smart people, diverse, energetic, welcoming
+
+Community, peer discussions, implementation stories
+
+Such a welcoming and loving community 
+
+Project health reporting. 
+
+Community-driven with real-world examples
+
+It is always improving and focusing on finding meaningful metrics to drive communities' health forward for a more equitable open source ecosystem (this is how I see CHAOSS <3)
+
+Augur really makes metrics highly visible on a large scale like I need
+
+Once set up, grimoirelab is a veeery impressive tool. I'm glad it's hosted under CHAOSS. 
+
+Standards
+
+Such an amazing community of lovely people