Create research strategy for measuring Community Health in L10N

[nukeador]

Foundational data gathering

• Leveraging D&I research what are the research questions we want to answer (@emmairwin Due: Feb 2nd )
• What goals/workflow do we have for conducting qualitative interviews? (@emmairwin Due: Feb 2nd )
• What data is available to us, and from what sources (@emmairwin /CHAOSS & Nebraska Due: Feb 2nd)
• Get stats from mercurial: Number of communities, activity level, contributors (translators, suggestions...). (@peiying2 @gueroJeff Due: Feb 2nd)

Additional background

• Consult Don (Project Debraska/CHAOSS) and Henrik (Community Dasboard) to understand current capabilities for measuring community health .
• Research external-facing research around localization in open source/ community health.

Health criteria

• Identify what problem statements are missing and propose final list. (@emmairwin and @nukeador)
• Create baseline metrics strategy that sets KRs for how we leverage the data and sources available to us, to answer research questions. (All)

Related taks

• https://github.com/mozilla/diversity/issues/53
• Process and select pilot participants #119 Process and select pilot participants

Working Documents

• Work notes
• Baseline Research Questions (workspace for collecting research questions)

[emmairwin]

First Meeting Document

• Schedule demo of Pontoon ‘trouble points’/ workflow (next meeting beginning)

@nukeador when do you want to actually 'measure' ? Deadline-wise?

[nukeador]

How long do we expect the research can take us? 2 weeks?

[emmairwin]

Depends on the methodology if we decide on dashboard approach only (and pontoon is available ) then not long, if we want to combine that with a survey, and maybe some quantitative I would say 3-4.

[gueroJeff]

If we went by a dashboard only approach, the data for a contribution dashboard will need to come from VCS, as it's the canonical source of truth and Pontoon's API does not apply to contributor data (see my comment on l10n data here: #118 (comment) ).

[nukeador]

@gueroJeff We have identified that we'll need to get the data from mercurial, as much as possible to identify communities, volume of activity and who's doing what. Ideally we would like to have this by the end of this week so we can inform the decision on participants selection.

I've added a checkbox about this and assigned to @peiying2 and you, let me know if that's OK.

[emmairwin]

@nukeador @gueroJeff here's an initial list demographics we want to capture, and research questions - the 'How' comes after we agree these are the right ones. We can also prioritize a bit later.

https://docs.google.com/document/d/1B-u2iFL1TSZbTts86bKN3tUDwqzhSmh97zpmdcUFgwk/edit#

Please add your own questions, especially around L10N specifically -which I know far less about.

[gueroJeff]

We also need stats from Github. We have a starting place from which to spot check where we're missing information.

Also, I'm happy to take this task myself, as Pei will be off Thursday-Friday.

[nukeador]

@emmairwin Today we talked about clearly define all dimensions we want to understand from pontoon+mercurial, both for localization communities and individuals.

@gueroJeff can provide a list of dimensions we can gather info about.

Cheers.

[Delphine]

Follow-up to our call. Was thinking: would bugzilla also be something we could look at (we have l10n stuff going on there). Like for example, responsiveness to open bugs, answering in timely manner... stuff like that?

[emmairwin]

@nukeador can you say more about what you mean 'define all dimensions we want to understand from pontooon+mercurial' ?

Here are the steps (I also created a document with goals here)

• List all research questions (things we want to learn)
• Identify all sources that can provide us with the information we need
• Identify gaps in sources (places we don't have information).

I think you mean the second step here, and that you are listing all the information available? So that might be Pontoon 1) locale, 2) username, 3) role
Where are you tracking that?

Action items for research questions in Slack.

[gueroJeff]

@emmairwin He's referring to all of the existing data points that can create a holistic picture of an individual localizer or a locale (e.g., volume of strings translated, frequency of contribution, etc.). The sources would be VCS, Pontoon, Bugzilla, GitHub.

@Delphine Bugzilla is definitely useful, but I wonder how we might track the dimensions there into something useful.

[emmairwin]

Ok perfect, where are we documenting that? Also the Community Dashboard /Bitergia - already has attributes for Bugzilla and Github, so we don't need to research that ourselves. When we're done this we should make sure we share share Pontoon, and VCS attributes back to that project so they can add them sooner.

…

On Tue, Jan 30, 2018 at 3:13 PM, gueroJeff ***@***.***> wrote: @emmairwin <https://github.com/emmairwin> He's referring to all of the existing data points that can create a holistic picture of an individual localizer or a locale (e.g., volume of strings translated, frequency of contribution, etc.). The sources would be VCS, Pontoon, Bugzilla, GitHub. @Delphine <https://github.com/delphine> Bugzilla is definitely useful, but I wonder how we might track the dimensions there into something useful. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#117 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADsyqSamKldQV3ZRRikJI_3qozsRdGKks5tP6IBgaJpZM4RhmBl> .

-- -- Emma Irwin

[gueroJeff]

It seems that we will be collecting this information here: https://docs.google.com/document/d/1B-u2iFL1TSZbTts86bKN3tUDwqzhSmh97zpmdcUFgwk/edit#heading=h.rtjoaaq1re47

When Axel and I last discussed adding l10n data to the community dashboard with Participation, we were told that it was too difficult to do this through VCS logs. Being that data in VCS is our most reliable, we don't have plans to expose localizer contribution data through the Pontoon API.

It seems that I don't have access to the Community Dashboard (tried through SSO and received an authentication error message). I'd like to see what data is in there for l10n. How do I gain access?

[nukeador]

I think we should focus on getting the data now for our consumption, have it on the dashboard can wait and be handled in the future #118

[emmairwin]

I need the research questions signed-off and prioritized by EOD Friday - before we can look at existing data.

NOTE: it is not important that we know where all the data is available YET - it’s OK if we have no idea where we will get data - it’s that we get the questions we are asking right. Don’t own the solution, own the problem” (quote from NASA talk yesterday :)

@nukeador can you please get consensus from the group on priority questions

https://docs.google.com/document/d/1B-u2iFL1TSZbTts86bKN3tUDwqzhSmh97zpmdcUFgwk/edit#

Once I have that, I will evaluate that list to ensure that d&I related insights are also prioritized as a finalized list.

[emmairwin]

Research questions (mostly finalized, last minutes still welcome: https://docs.google.com/document/d/1B-u2iFL1TSZbTts86bKN3tUDwqzhSmh97zpmdcUFgwk/edit#

Document for tracking data sources https://docs.google.com/spreadsheets/d/1PQSbTbDXk1_cyhkP-dXwFdfPdnL6Yolvtg_5SI7NAPI/edit#gid=0

Interview best practices is still forming, I am going to move it to diversity repo though... https://wiki.mozilla.org/Innovation/Projects/Community_Development/Community_Standards/Diversity_Inclusion/Data_Metrics#Conducting_the_Interview

Will talk to Henrik last week, consulted Don this past Tuesday for input into research questions

[gueroJeff]

Closing due to inactivity for an extended period.