Migrate option documentation to nixosOptionsDoc

[emilazy]

Description

Okay, here we go!

These changes move option documentation generation to nixosOptionsDoc and migrate all the existing documentation to Markdown. The output given the same input is almost identical or in some ways improved (e.g. no <quote> needed for smart quotes); the most notable non-positive changeis some unstripped empty lines at the end of literalExamples; these can also be seen in the NixOS manual and should be addressed upstream. All automatic conversions were verified by nix-doc-munge to produce no significant changes in output, and almost all manual conversions resulted in equivalent or improved semantics and formatting; a very small number had to be slightly downgraded (e.g. tables, multiple terms for one definition in a definition list), mostly due to easily-fixable things like "nobody added processing for the relevant Markdown node to the options documentation part of nixos-render-docs yet".

I won't go into justification too much, as it was already discussed extensively in #4158, but I will summarize that I think using the upstream NixOS tools will be more robust in the long run and make sharing code and contributors with NixOS easier. I admit that I prefer AsciiDoc to Markdown myself, but the benefits of using the same format and tooling as NixOS far outweigh that, and anyway NixOS's Markdown has added almost all of the semantic niceties you would want regardless. (And, of course, writing DocBook by hand is the worst of all options; in many cases I had to manually mark options as Markdown because they already contained Markdown syntax that wasn't rendering correctly but was much less unwieldy to write than the correct DocBook would have been.)

The commits have fairly detailed explanations, but the overview is:

• Pin the last version of nixos-unstable where nixosOptionsDoc still supported DocBook.
• Start using it and set up some infrastructure for the conversion.
• Manually tweak or convert some options that can't be handled automatically.
• Convert the rest with my version of nix-doc-munge.
• Remove the pin.
• Strip out all the lib.mdDoc calls, as all documentation is interpreted as Markdown in the latest Nixpkgs.

The last part could be omitted if desired; I chose reducing visual noise over ease of backporting, as the release branches of Home Manager don't seem to get many changes anyway. The presence of lib.mdDoc calls could also mislead people into thinking that unmarked documentation will still be interpreted as DocBook. Still, I can undo that change if it's not preferred.

The following commits were automated, whether by simple textual replacement or nix-doc-munge:

• e04de5b treewide: mkPackageOption -> mkPackageOptionMD
• b5a65b9 treewide: mkAliasOption -> mkAliasOptionMD
• c1d8d2a treewide: adjust some DocBook for conversion (mostly; a few trivial "human-powered sed" tweaks)
• 36a53d9 treewide: convert all option docs to Markdown
• 9f9e277 treewide: remove now-redundant lib.mdDoc calls

The rest are all manually written and hopefully not too difficult to review.

This is only the first half of the migration: it still relies on the deprecated optionsDocBook output (hence the warnings when evaluating under 23.11) and has the full DocBook toolchain for manual generation. I have an almost-complete full conversion of the manual locally that preserves the content and styling, but there a few things that need addressing in upstream nixos-render-docs before it's ready to PR (mostly just some minor rendering issues and a couple additional customization points), so that's probably a few weeks off as I'll need to PR the changes upstream.

I tested all commits with the following:

emily@yuyuko ~/D/home-manager (option-docs-migration)> cat >test.bash
#!/usr/bin/env bash
set -eu
if uname | grep -q Darwin && ! grep -q nixosOptionsDoc docs/default.nix; then
	# Work around Nix sandbox bug on Darwin.
	nixpkgs=github:emilazy/nixpkgs/841136098f0065ebdd7121646165aedefb31e819
	legacyArgs=(--arg pkgs "import (builtins.getFlake \"$nixpkgs\") {}")
	flakeArgs=(--override-input nixpkgs "$nixpkgs")
else
	export NIX_PATH=nixpkgs=flake:nixpkgs/$(nix flake metadata --json | jq -r .locks.nodes.nixpkgs.locked.rev)
	legacyArgs=()
	flakeArgs=()
fi
set -x
./format -c >/dev/null
nix build "${flakeArgs[@]}" --no-link .#docs-{html,json,manpages}
nix-build "${legacyArgs[@]}" --quiet --no-out-link -A docs.html -A docs.manPages -A docs.json >/dev/null
nix-build "${legacyArgs[@]}" --quiet --no-out-link tests -A build.manual >/dev/null
emily@yuyuko ~/D/home-manager (option-docs-migration)> nix shell nixpkgs#bash -c git rebase --exec 'bash test.bash' upstream/master

and I checked that all tests pass successfully on the final commit.

I am happy to be pinged for any future issues with documentation processing and generation going forward.

P.S. It's not too hard to update this PR, as the most invasive changes are all automated, but it is a little fiddly, so I'll hold off on updating it for any other changes until this is considered ready for merge (and would appreciate holding off on merging any others from that point to the final merge). It will, of course, cause conflicts with many open PRs on merge, especially those that touch documentation, and PRs after merge will have to be checked for containing DocBook documentation, but that's unavoidable and should be simple to resolve for any given PR.

Checklist

• Change is backwards compatible. (for users, not contributors)

• Code formatted with ./format.

• Code tested through nix-shell --pure tests -A run.all or nix develop --ignore-environment .#all using Flakes.

• Test cases updated/added. See example.

• Commit messages are formatted like

  {component}: {description}
  
  {long description}
  

  See CONTRIBUTING for more information and recent commit messages for examples.

• If this PR adds a new module

  • Added myself as module maintainer. See example.

Maintainer CC

cc @rycee @ncfavier

[rycee]

Thanks and much appreciated! Will try to look through this as quickly as possible.

[rycee]

About the

Strip out all the lib.mdDoc calls, as all documentation is interpreted as Markdown in the latest Nixpkgs.

change, I'll have a think. I'm leaning towards it being fine to strip them out but, like you say, it makes backports a little more involved.

And to be clear, it is OK to merge new modules and such while this PR is open? We can consider the docs directory frozen until the merge but it would be a pity not to be able to merge module changes as usual.

[emilazy]

Yes, that's totally fine of course; my intention is to leave this PR as-is during review, and then after it's done and this is considered ready to merge, I'll run the conversion one last time (and handle any options that don't convert automatically) and rebase the PR accordingly. I'd just appreciate it if I could have couple days of no merges at that point so I'm not having to play catch-up with master to update it :)

(FWIW, I suspect backports will be a bit manual either way, as anything adjacent to or touching documentation already likely won't be able to be automatically cherry-picked.)

[emilazy]

Naturally I forgot the most ironic possible thing: updating the documentation. (Perhaps it could use a release note too? But it shouldn't be relevant to any user unless they're maintaining a custom module.)

[rycee]

I've looked through the commits now and checked the resulting output. Impressive work, looks very good in general! As far as I can tell the only negative is the lack of table support but I think using the description lists is a decent compromise.

I think we are good to merge this whenever you wish. From when would you like the PR merge freeze?

[ncfavier]

Tested this branch, builds and looks fine.

Are these warnings expected (to go away)?

trace: warning: optionsDocBook is deprecated since 23.11 and will be removed in 24.05

[emilazy]

I've pushed some changes per the review comments. Holding off on the final rebase until it's time.

I've looked through the commits now and checked the resulting output. Impressive work, looks very good in general! As far as I can tell the only negative is the lack of table support but I think using the description lists is a decent compromise.

I think we are good to merge this whenever you wish. From when would you like the PR merge freeze?

Yeah, the table support is a shame, but all the machinery is there upstream, it just needs plugging into option docs. I plan to do that but didn't want to hold this up for it. As far as freezing goes, I'm flexible; should be able to get this finished within 2-3 days of ping (and probably less) any time next week, since the actual work is simple. If there are any other big PRs that touch/add documentation that are approaching ready for merge it might be good to get those in first to save other people the conversion effort.

Tested this branch, builds and looks fine.

Are these warnings expected (to go away)?

trace: warning: optionsDocBook is deprecated since 23.11 and will be removed in 24.05

Yeah, this is expected and unfortunately unavoidable for now; we need the DocBook options output that Nixpkgs is getting rid of in order to plug into the existing nmd/DocBook pipeline. From the PR message (which is super long, so I don't blame you for missing it):

This is only the first half of the migration: it still relies on the deprecated optionsDocBook output (hence the warnings when evaluating under 23.11) and has the full DocBook toolchain for manual generation. I have an almost-complete full conversion of the manual locally that preserves the content and styling, but there a few things that need addressing in upstream nixos-render-docs before it's ready to PR (mostly just some minor rendering issues and a couple additional customization points), so that's probably a few weeks off as I'll need to PR the changes upstream.

The next steps will be me making some changes to nixos-render-docs upstream to adapt it to the Home Manager (and nix-darwin) context, and then I can finish the job off and we'll have a zero-DocBook documentation generator. Might take me a little bit to get around to, but certainly before the 23.11 release.

[rycee]

Sounds excellent, then I propose to start the PR merge freeze on July 19 (Wednesday) with the hope that we can merge this before the end of the week.

I checked the repo settings and it seems like it is possible to lock a protected branch and as far as I can tell that will prevent any commits from being added to the branch. So on the morning of 19th (central European time) I'll mark the master branch as locked and add a comment in this PR. Unless some serious issue needs to be resolved (I'm mainly thinking a security or data integrity issue) then I will keep it locked until @emilazy writes a comment saying it is OK to merge.

I'll make an effort to be available in the Matrix chat during this time, at least during CEST working hours, when I'm on my computer anyways 🙂

[emilazy]

Wednesday sounds good to me! I don't expect it will take long at all to update the PR; just a matter of being around to run the scripts and tests and handle any new option documentation needing manual conversion. I've rebased against latest master to get a head start on that and it was purely mechanical. I also added a note of this change to the release notes in case people need to update documentation in third-party modules (currently the local manual would not include them anyway, I believe, but as this could change in the future it's good to make sure people are using the right syntax).

[rycee]

@emilazy Great, then I'll lock the master branch tomorrow and you should be safe to do the migration without conflicts.

@ncfavier, @sumnerevans, @teto Please mind that @emilazy will be migrating the documentation to the Nixpkgs documentation generator starting tomorrow Wednesday. To avoid merge conflicts, please hold off on doing merged until this PR is complete. I'll try locking the master branch so I think it won't be possible to merge anything even if you try but I'm not fully certain that it works like that.

[teto]

acked ! Sounds like huge work. Thanks !

[emilazy]

FWIW as long as there's no commits to master past 2f84579 then this is good to merge as-is. Otherwise I'll update it ASAP of course.

[rycee]

I locked master now. If I understand @emilazy correctly, then the PR is already up-to-date. If that is the case then we can merge in the evening ~10-15 hours from now.

If anybody wants to test and comment on this PR then now is the time 🙂

[emilazy]

Yep, should be good to go!

[rycee]

Ok, had a look at the latest changes, the generated documentation, and the release notes. All looked just fine to me so I've merged now. Thanks a lot for the hard work @emilazy, much appreciated!

I'm sure there will be some fallout over the next few days/weeks with people having failed builds due to having their own modules with docbook descriptions. So please be mindful of this and let them know that they can change the descriptions to the markdown. Can point them to the release notes.

[emilazy]

Wonderful, thank you for getting this reviewed and merged! I will be busy for the next short while but will try and work on nixos-render-docs when I can.