RFC: restructure website content, roughly following Diátaxis

[h-vetinari]

There are several ongoing efforts to improve the conda-forge website, which has yielded the new website, status page, as well as some work around the infrastructure pages.

Additionally @jaimergp wrote in the context of the more tightly-scoped #2154:

Last year, I experimented with how Diátaxis would apply to our docs in https://cf-infra-docs.netlify.app/docs. Some of the docs written for that experiment are being migrated over to our new website. @zklaus is almost finished, I think, with that effort.

I took that idea and came up with a rough draft of how the website could be structured (originally looking only at the infrastructure page, but I felt I needed a big picture view even for that). Some new content would have to be written, but mainly it is intended to reshuffle existing content into easily discoverable and more granular pieces.

As Diátaxis intentionally aims at gradual transformation rather than big bang refactoring, this outline would likely change as we iterate, but I wanted to get people's thoughts on the rough strokes (and then discuss it in the core call), so we can agree on the overall direction and let people start chipping away at some first pieces.

This is based on looking at our current docs as well as @jaimergp's experiment with Diátaxis; a more detailed comparison is available below the fold.

Welcome
For users
└ ...                               # not in scope here
For maintainers
├ Tutorials
|  ├ Generate a conda recipe        # Using greyskull or whatever
|  ├ Write your first recipe        # Segue from first tutorial, make some changes
|  ├ Submit your first recipe       # Segue again, process for submission
|  └ ...
├ How-to guides
|  ├ Easy                           # the below is just a selection, obviously
|  |  ├ Read a recipe               # Component overview, refer to in-depth explanation
|  |  ├ Rerender feedstock
|  |  ├ Test packages
|  |  ├ Deal with numpy
|  |  ├ Enable osx-arm64            # Same for arches
|  |  ├ Talk to the bots            # refer to bot command reference page
|  |  └ ...
|  ├ Medium
|  |  ├ Cross-compile
|  |  ├ Enable CUDA
|  |  ├ Multi-output recipes
|  |  ├ LTS branches
|  |  └ ...
|  └ Emergencies
|     ├ Patch repodata
|     └ Mark a package broken
├ Understanding conda-forge         # Explanation part of Diátaxis
|  ├ Life-cycle of a package        # essentially https://cf-infra-docs.netlify.app/docs/fundamentals/life-cycle/
|  |  └ ...
|  ├ How a package gets built
|  |  ├ Compilation concepts        # shared/static/etc.
|  |  ├ Environment roles           # e.g. build: vs. host:
|  |  ├ Compilers                   # e.g. role of activation
|  |  ├ Global pinning
|  |  ├ Run exports
|  |  ├ Jinja functions
|  |  └ ...
|  ├ How our bots work
|  |  ├ Migrations
|  |  ├ regro infrastructure
|  |  └ ...
|  ├ Security
|  |  ├ Output validation
|  |  └ ...
|  ├ Maintainer roles
|  └ ...
├ Reference
|  ├ Infrastructure                 # more in-depth discussion in #2154
|  |  ├ Staged recipes              # including teams
|  |  ├ Compilers
|  |  ├ Other core packages         # sysroots, CDTs, msys2, ...
|  |  ├ CI Setup                    # docker images, conda-forge-ci-setup
|  |  ├ CI providers
|  |  ├ Global pinning
|  |  ├ Feedstock evolution         # conda-smithy
|  |  ├ Feedstock metadata          # graphs, feedstocks, feedstock-outputs
|  |  ├ Metadata corrections        # repodata-patches, admin-requests
|  |  ├ Bot repos                   # all things regro
|  |  ├ Website
|  |  └ ...
|  ├ Bot commands
|  ├ Feedstock settings             # conda-forge.yml, CBC, migrators
|  ├ Glossary
|  ├ Current pins                   # Auto-generated / subset?
|  ├ Ongoing migrations             # refer to status page
|  └ ...
├ Miniforge
└ FAQ                               # separate section because questions cover both how-to & reference...?

Comparison with current docs and Diátaxis experiment

	vs. docs	vs. Infra Exp.	Comment
Welcome	== here	== Welcome
For users	== here	n/a	could surely be improved...
└ ...			... but not in scope here
For maintainers	== here	n/a
├ Tutorials	partially here	== Tutorials
| ├ Generate a conda recipe			Using greyskull or whatever
| ├ Write your first recipe			Segue from first tutorial, make some changes
| ├ Submit your first recipe		~= placeholder	Segue again, process for submission
| └ ...
├ How-to guides	distributed all over	== How-to guides
| ├ Easy
| | ├ Read a recipe			Component overview, refer to in-depth explanation
| | ├ Rerender feedstock
| | ├ Test packages
| | ├ Deal with numpy	roughly here
| | ├ Enable osx-arm64	roughly here		Same for arches
| | ├ Talk to the bots			refer to bot command reference page
| | └ ...
| ├ Medium
| | ├ Cross-compile	~= here
| | ├ Enable CUDA	~= here
| | ├ Multi-output recipes	~= here
| | ├ LTS branches	~= here
| | └ ...
| └ Emergencies
| ├ Patch repodata
| └ Mark a package broken	~= here
├ Understanding c-f	not much	~= Fundamentals	"Explanation" part of diataxis
| ├ Life-cycle of a pkg		== here
| | └ ...
| ├ How a pkg gets built
| | ├ Compilation concepts			shared/static/etc.
| | ├ Env. roles	short comment		e.g. build: vs. host:
| | ├ Compilers	partially here		e.g. role of activation
| | ├ Global pinning	~= here
| | ├ Run exports	~= here
| | ├ Jinja functions
| | └ ...
| ├ How our bots work		see here
| | ├ Migrations	~= here
| | ├ regro infra
| | └ ...
| ├ Security	~= here	see placeholder
| | ├ Output validation
| | └ ...
| ├ Maintainer roles	roughly here & here
| └ ...
├ Reference	all over	== Reference
| ├ Infrastructure		~same content split somewhat differently
| | ├ Staged recipes		roughly here	incl. teams
| | ├ Compilers	roughly here
| | ├ Other core packages			sysroots, CDTs, msys2, ...
| | ├ CI Setup			docker images, conda-forge-ci-setup
| | ├ CI providers		roughly here
| | ├ Global pinning	~= here		+explain migrators
| | ├ Feedstock evolution			conda-smithy
| | ├ Feedstock metadata			graphs, feedstocks, feedstock-outputs
| | ├ Metadata corrections			repodata-patches, admin-requests
| | ├ Bot repos			all things regro
| | ├ Website		== here
| | └ ...
| ├ Bot commands	~= here	== here
| ├ Feedstock settings	mostly here	== here	conda-forge.yml, CBC, migrators
| ├ Glossary	== here	== here
| ├ Current pins		== here	Auto-generated / subset?
| ├ Ongoing migrations			refer to status page
| └ ...
├ Miniforge		== here
└ FAQ	~= here		separate section because questions cover both how-to & reference...?

[jaimergp]

Love this. The biggest concern is not breaking any existing URLs and anchors that might refer to existing content.

How we have addressed this so far:

• If there's a one-to-one correspondence between URLs (the same page with the same anchors just moved from one place to other), we can use the redirects defined in docusaurus.config.js.
• If different sections have moved to different pages, the original page can remain in place as an unlisted page with the old anchors pointing to the new URL and section. Example at https://github.com/conda-forge/conda-forge.github.io/blob/main/src/pages/announcements.md.

Another point is how sometimes the line between "For users" and "For maintainers" is not entirely clear. At least, I struggled while categorizing it during that experiment in the draft site. Maybe because I was trying to shuffle existing docs around, and didn't leave room for new content.

In that vein, if categorizing things in the filesystem is too restrictive, we can also leverage the custom sidebar setup (the paths and the categories match in this case, but they don't have to), and the document tags to label pages with certain information (for-beginners, easy, advanced, whatever).

cc @zklaus for awareness

[h-vetinari]

The biggest concern is not breaking any existing URLs and anchors that might refer to existing content.

Yeah that would likely be a large part of the challenge - for now I was happy to draw outlines unencumbered by that though 😅

Another point is how sometimes the line between "For users" and "For maintainers" is not entirely clear. At least, I struggled while categorizing it during that experiment in the draft site.

When I was looking at this, I kind of saw the utility of the existing separation, i.e. someone wanting to just consume packages has a very different set of concerns than someone trying to maintain a feedstock. Consequently, I had descoped the "for users" part in my draft, but I agree with you that the separation is not obvious, and likely harmful in terms of having an easy "on ramp" for turning package consumers into conda-forge contributors. It would be nice if we can smoothen that transition somehow.

[zklaus]

[...] having an easy "on ramp" for turning package consumers into conda-forge contributors. It would be nice if we can smoothen that transition [...].

My thoughts exactly!

[zklaus]

Out of a discussion with Jaime: It would be nice to have a single overview page explaining compilation on Windows, i.e. ms vs clang vs mingw, why and how.

[jaimergp]

I think it's time to move forward with this. This question remains, though: how do we even start?

• 1. Feature branch where we slowly move to Diátaxis and one day we merge to main? Risk of documentation drift if we take too long, and painful conflict managing, but very convenient and not obtrusive.
• 2. Slowly populate main with Diátaxis structure. Two ways:
  • a. Create a new top-level Documentation section (e.g. /diataxis) where we populate things from scratch. This will lead to duplicated content, and tons of redirects when we finally move things to /docs.
  • b. Adapt /docs little by little. A bit more involved and confusing for readers, but I think this is what Diátaxis proposes anyway. We would create a new category in the sidebar with the proposed structure (I think we can link to the same pages from different categories with no issue), and then one day we promote it to top-level.

I'm leaning towards strategy (2b) and will propose so in a draft PR so you can see what I mean.

[jaimergp]

Opened #2633, with an approach I think it's even better. We don't need to use the same sidebar, we can use a separate one within the same Docs section! We can even link to common pages using ref items.