Extract the development docs from this repository

[tacaswell]

We have historically built all of our documentation from the sphinx source in the main matplotlib/matplotlib repo (this one). We would then host this by unpacking the result of the build at the top level of website + into a version specific sub-folder. We would also rely on

While this served us well for over a decade, over the past few years we have slowly started to pull things apart. The goal is to ultimately de-couple the version independent parts of the documentation (which should be re-published as soon as they are update) from the version-specific docs (which should be Steps we have already taken so far:

• move the cannonical docs to https://matplotlib.org/stable https://discourse.matplotlib.org/t/canonical-documentation-have-moved/21863
• split the third-party docs out into its own repo https://github.com/matplotlib/mpl-third-party
• have a sphinx theme for all of our websites https://github.com/matplotlib/mpl-sphinx-theme
• split the top-level index into its own repo https://github.com/matplotlib/mpl-brochure-site
• have built docs for the governance https://github.com/matplotlib/governance
• moved our ground-truth hosting to digital ocean https://github.com/matplotlib/matplotlib.org

The next thing we should pull out into its own repository is the content under https://matplotlib.org/stable/devel/index.html . We probably should copy-paste the GHA/sphinx configuration from the governance repo and update the rst in that framework.

We should do this as a second step after we finish @noatamir 's update of that content (to make it easier to review the changes at each step).

[QuLogic]

Do we want to put this in mpl-brochure-site or a separate repository?

[tacaswell]

Originally I thought mpl-brocure-site, but I now I think a separate repository. We already have a whole bunch of repos serving docs so we might as well lean into that and separate the repos by purpose.

[timhoffm]

Originally I thought mpl-brocure-site, but I now I think a separate repository. We already have a whole bunch of repos serving docs so we might as well lean into that and separate the repos by purpose.

Makes sense. It does not fit into mpl-brochure-site.

We may need a place to document which repositories are used for which parts of the docs. - Should this be in a separate repo as well? 😜

[noatamir]

Maybe just in the docs themselves?
An overview of the structure/architecture of the docs would fit in the docs no?! it's a bit meta but makes sense in a way...

[story645]

An overview of the structure/architecture of the docs would fit in the docs no?!

Could maybe go in the dev guide for doc contributions since "where is this thing I want to change?" is a not infrequent new contributor question.

[jklymak]

I personally think we should have as few repos as possible. i.e. one tied to releases, and the other not tied to anything. mpl_third_party is a bit different as it has its own review and build pipeline.

[tacaswell]

Currently we are pulling the artifacts for our docs from 14 different repos (see
https://github.com/matplotlib/matplotlib.org/blob/e57c891aaddf8fda1c5fd06dcb5c99875e2c05bf/matplotlib.org.yml#L10-L24) so although I agree with the sentiment, I think we are well past hoping to keep the number of repositories involved to very small numbers. The very large number of repos contributing to the docs is not new, we were previously exploiting the implicit layering that github pages so it was much harder to see!

If we focus just on the core library docs we have:

• matplotlib.github.com : this is the built output of our versioned docs + cmocean docs. Everything in here should be strictly output of build scripts. We should not really do PRs (maybe should just turn them off?) and probably want to lock down write access to the release managers?
• devdocs : this is built and published on merge to main. This should never get PRs, the push process force-pushes so we only ever have one commit (😱). We should probably disable PRs on this repo and only leave write access to the deploy key?
• goverance : in addition to holding the governance this now has a build and publish pipeline for a rendered version of the governance. This repo should be pared down to just be the governance (process and people)+ its build pipeline and write access restricted to the steering council
• mpl-brochure : this is just the top-level index.html + its build pipeline. This probably should also have write permissions restricted to a small set of people (release manager + ?)
• cheatsheets : There are (as the name says) the cheatsheets. It has its own build process and Nicolas is responsible person for this repo, we should defer to him about what the write permissions should be
• matplotlblog: this should probably be replaced with a re-direct to the https://blog.scientific-python.org/tags/matplotlib/ at the caddy level and archived (that is a separate issue though)
• mpl-third-party : the third-party page which has a very custom build (like cheatsheets). I'll defer to @jklymak about what we want the write permissions on this to be
• (proposed) mpl-devel-guide : the content being proposed to be pulled out here. I think we want the write permissions on this to exactly match the write permissions on the main repo. This is both to keep the level of write access the same as it is now and because I think that everyone we trust to merge code to the library we should trust their judgement to document how to work on it!

I've been thinking about security for unrelated reasons recently. I am not concerned about what people we know and have given access to are going to do, however I am concerned about what someone who gets control of one of our accounts might do which is why I want to restrict write permissions on the goverance and very repos that are the source for the high-traffic parts of our website.

I do not think there is anything we really want to merge together, we are already paying the complexity cost of "more that one" at the caddy configuration level, and wanting to have different review/merge rules on https://matplotlib.org and the development guide I think we should go with (just) one more repo and include the above in the devel guide as a part of how to work on the docs.

[jklymak]

and wanting to have different review/merge rules on https://matplotlib.org/ and the development guide I think we should go with (just) one more repo and include the above in the devel guide as a part of how to work on the docs.

OK, that makes sense. maybe just leave it open to be more than just "dev guide" in case we want other docs not tied explicitly to the release cycle?

[tacaswell]

OK, that makes sense. maybe just leave it open to be more than just "dev guide"

That seems correct to me, but no better term is coming to mind quickly. I would also want to move the communication guidelines out of governance to the new repo along with guides/templates on how to handle common situations with (new) contributors ("we do not assign issues, just start working", "congrats on your first PR, hope to here from you again!", "this better handled on discourse", "can you please include a minimal running example + version numbers", "your local installation is broken, uninstall everything and try using just 1 (one) package manager", etc), the triage guide, the PR review guide, etc, the meta-doc on docs, etc.

In our usual way, https://matplotlib.org/stable/devel/index.html is devel in the source tree, "Develop" in the site header, and has an H1 of "Contributing" (with a nested H2 that is also "Contributing").

"contributing-and-maintaining-and-community-guide" is too long.

[story645]

Cause this is coming up in #27386, over the past year the the dev docs have been updated and gitwash stripped out (thanks @melissawm!). I'd prefer #27265 to go in before we move things (especially since it includes a 'where are docs section' I can flesh out).

maybe just leave it open to be more than just "dev guide" in case we want other docs not tied explicitly to the release cycle

Broadly speaking, the version independent docs are consolidated to:

• https://github.com/matplotlib/matplotlib/tree/main/doc/devel
• https://github.com/matplotlib/matplotlib/tree/main/doc/users/project
• https://github.com/matplotlib/matplotlib/tree/main/doc/users/resources

The plan for resources was/is that it should get something like the third-party-package repo so it sort of doesn't matter, and I think a follow up PR to a migration is updating cross links using intersphinx (which there's a configuration we can use to treat it as one big doc so none of the links break so we may not even need to).

ETA: the users/project docs are also in a weird space in that for the most part I think the contents of those docs should be very difficult to change - it's mission/code of conduct/licence/history/credits - w/o major discussion. Like our CoC or mission shouldn't be changeable by way of one approval unless it's like a typo/formatting fix.

[jklymak]

I don't think we need to move anything - simply point /stable/devel to /devdocs/devel, etc for other things that should be current. Or if we don't want folks popped into devdocs, just automatically rsync the subdirectory into /stable whenever /devdocs is updated. When this last came up, I don't think we had the possibility of server side redirects.

[jklymak]

Ping @QuLogic for an opinion on if either of the above are doable.

[timhoffm]

I feel splitting dev instructions out into a separate repo decouples them too much, and adds extra burden (separate repo to configure and handle).

Primary requirement: The dev instructions should be updated contiuously. Typically dev happens on main and instructions how to work there are relevant, not the dev instuctions that were active during the latest release.

I see two possible ways to achieve this:

• If @jklymak's proposal is viable, it seems like a low-effort change.
  Advantages:
  • No changes for the authors. They can work as if it's still one doc.
  • No additional configuration for doc builds.
• If not, I propose to keep the dev instructuion docs in this repo as an independent doc; i.e. in a separate folder with separate configuration. This gives us most of the freedom of a separte repo, but simplifies usage and maintanance:
  • authors can still work in a single repo
  • requirements can be shared
  • when config updates are necessary, it's much simpler to change two files in one repo, compared to two files in two repos.
    IMHO the additional aspects we get from a separate repo, like different permissions, are not relevant here.

[QuLogic]

On the server, there are two options:

1. A redirect, which will point the browser to the new location.
2. A rewrite, which will return the new file content from the old location.

We would need to be a bit careful if devdocs renamed/moved a file compared to stable, though we do try to insert redirect HTML when we do. The other downside is that then all other links are to devdocs.

I don't think rsync is viable, as devdocs is in a separate repo from the versioned docs, and both sides are git checkouts.

[story645]

I think setting a couple of the high profile links to contribute to the dev docs may catch most of the cases if we want something on the simple side:

• navbar 'contribute' link
• docs landing page 'contribute' link
• install warning admonition link

[jklymak]

On the server, there are two options:

1. A redirect, which will point the browser to the new location.
2. A rewrite, which will return the new file content from the old location.

Probably a redirect is more appropriate, unless all the links in devel are relative, which I doubt they are.

We would need to be a bit careful if devdocs renamed/moved a file compared to stable, though we do try to insert redirect HTML when we do.

I think we need to be careful to do this anyways, so I don't think thats too strong an objection.

The other downside is that then all other links are to devdocs.

I think that is OK, and a strong argument for a redirect versus rewrite. If you want to go to devel for some reason, it's not too likely you are then likely to get confused if you are not on stable/examples anymore. Whereas clearly stable/devel is confusing.

[QuLogic]

Here is the followup PR from the meeting discussion: matplotlib/matplotlib.org#36