Carry out research about pip documentation

[ei8fdb]

Research questions:

i.e. what do we want to know?

• When pip users have a problem or experience an error, what do they do? (note: crossover with Carry out research to understand who uses pip #8518)
• When pip users want to accomplish a specific task, what do they do? (note: crossover with Carry out research to understand who uses pip #8518)
• How do users use pip's current documentation?
  • Does it provide good orientation / is the structure clear?
  • Does the documentation look "official"?
    • Does it look trustworthy?
    • Does it look like an "official" Python property?
  • Do users understand the terminology?
• How do users distinguish between pip documentation and other documentation in the packaging ecosystem (e.g. packaging.python.org, PyPI help page, etc.)?
• How do pip/packaging maintainers think about documentation and write documentation?
  • Is it obvious where documentation should go?
  • What ideas do they have about improving the docs?
  • Are there (written or unwritten) documentation standards?

Preliminary research plan

i.e. how are we going to find all that out?

Note: We should expect some foundational insights from @ei8fdb's work on Carry out research to understand who uses pip

Some initial ideas:

• Investigate if we can use analytics to draw conclusions about pip's docs (e.g. time on page, etc.)
• Gather feedback while users are on pip's documentation (e.g. allow users to send feedback in real time)
• Conduct user interviews with a range of pip users
• Ask users to participate in journal studies
• Collect feedback from the community via surveys
• Conduct interviews with packaging maintainers

Resources

i.e. where can we go for help?

• Divio documentation system: breaks down documentation into 4 segments: tutorials, how-to guides, technical reference and explanation
• Google technical writing course
• Write the docs resources
• How to test the usability of documents
• Information architecture basics

People

i.e. who can help us?

• @evildmp

  • Review research plan
  • Review outputs / recommendations (see below)
  • Periodic involvement as docs are rebuilt

• @jdittrich

  • Assist in collecting and analysing data
  • Note: Jan has an academic interest in how people understand documentation (PHD subject)

Outputs

i.e. what will we have at the end of all this?

1. Documentation about pip's documentation - how it is used, it's shortcomings, etc.
2. Strategy for how to restructure / rewrite pip's documentation
3. Guide for how to contribute to pip's documentation, outlining:
   • documentation best practices / principles
   • copywriting styleguide / glossary of terms

Questions for the team

To be discussed with the pip maintainers

• Which research questions (from the list above) are most important?
• What is the best way to involve the team when working out the strategy for restructuring / rewriting the docs?
  • Should we run a workshop on this?
  • Should UX periodically present our ideas for feedback?
    • How often?
    • What channel? (team meeting, HackMD review, GitHub, other?)
• How can we put in place the resources / people to execute the strategy?
  • Is it within the scope of the currently funded work?
  • If not, how can we move forward on this?

[ei8fdb]

Adding this here after @pradyunsg tagged me and others in PR #9146.

My first question is what questions do we want to answer with analytics? Gathering ideas on what we want to find out would help us decide.

Here are my thoughts -

To understand how users are using the documentation

The following data is commonly gathered in order to understand how documentation is used :

• what pages are most visited? (e.g. top 10/100)
• how long do users spend on each page?
• inbound referrers
• outbound links
• what paths users take through the documentation

With the above data we would be able to understand how the documentation is used, decide on what pages need to be updated frequently, what pages need to be tested with users, what pages can be removed/rationalised.

Helpful, but needs further thought

The following information would be helpful to have, tho' would need to be discussed to make sure it was implemented in the right way.

• the language the user speaks or the coarse geographic region the user has visited from

This would give us rough understanding of the language the user may speak - useful when deciding on translation.

• the coarse device type information - mobile/tablet/laptop/desktop/other device

This would be useful to understand about information layout

Consent and transparency

My opinion would be analytics would be opt-in for the documentation visitor. Explaining why we requst permission to gather the data should be communicated.

It would also be important to be transparent with the community about what type of data we are gathering, and what it is being used for.

I'd like to suggest discussion about making the above data public. It would be important to make sure we were not gathering IP address, granular geographic information. This would show the community what data is generated, and (I would think) be interesting for the community to see.

[pfmoore]

For some reason I can't apply a 👍 emoji to the above comment, so I'm putting it here.

[pradyunsg]

It was a locked issue, which, doesn't allow reactions.

[ei8fdb]

It was a locked issue, which, doesn't allow reactions.

I've unlocked it now. Don't remember why... 🤔