nixos/make-options-doc: replace xslt with python where possible, remove olinkdb #212289
Conversation
this doesn't seem to be set anywhere, not from make-options-doc nor anywhere else.
pennae
changed the title
this restores mergeJSON to its former glory if…merging json, and extracts the MD rendering into a new script that will run instead of the py+nix+xslt pipeline we previously ran to convert options.json to docbook. this change alone gives a noticable performance boost when building docs (18s instead of 27s to build optionsDocBook). no changes to rendered output, except for a single example in the rsnapshot module that uses hard tabs for indentation instead of spaces. this probably isn't important. docbook warnings remain with mergeJSON since the other processing steps output single files instead of directories. since we'll only keep the check until 23.11 this is probably also not important to fix. also contains a few improvements to error reporting in the MD renderers.
as far as we can tell nixos has only ever had a total of one olink, and currently has no olinks at all. we can't currently represent olinks in markdown docs, and if we re-add support for cross-document links they will take a different form (and not use docbook, which will have to be phased out before we re-add anything). the olinkdb is thus unused and takes 10 seconds on our machine to build, holding up the rest of the manual for no benefit.
pennae
force-pushed
the
optionstodb-py
branch
from
f2209a2
to
fbb7830
Compare
|
This would break the table of contents in flake.parts. I'm looking into what causes that.
At least it lets me generate a table of contents easily, and manipulate the structure of the generated document. Doing that without docbook is going to be painful, so I'm glad docbook is still involved - for the time being. |
|
Pushed a fix. Wouldn't have been necessary with a proper xml library, but maybe that's ok. |
the rendering tool this will all become in due course can have many output formats. we could keep docbook export support around for probably quite a long time, but we'd rather not use it as an intermediate for other outputs if at all possible. ideally we'd add converters targeting other formats that have all the necessary features, but aren't quite as far from the in-tree inputs. |
NixOS/nixpkgs#212289 removes the xls files we relied on in favor of a new solution that goes from options.json to docbook using a python script. This commit is a band-aid approach of pulling in the necessary files from an old nixpkgs revision to fix the build of the manual until we can solve it properly. Since the intention of upstream is to get rid of docbook for good, it probably makes sense to port our module documentation to markdown already when porting our manual build process to the new toolchain.
NixOS/nixpkgs#212289 removes the xls files we relied on in favor of a new solution that goes from options.json to docbook using a python script. This commit is a band-aid approach of pulling in the necessary files from an old nixpkgs revision to fix the build of the manual until we can solve it properly. Since the intention of upstream is to get rid of docbook for good, it probably makes sense to port our module documentation to markdown already when porting our manual build process to the new toolchain.
- json.dump(convertMD(unpivot(options)), fp=sys.stdout)
+ json.dump(unpivot(options), fp=sys.stdout)@pennae, this is a breaking change for the commonmark and asciidoc generators. Much like the docbook generator, these worked on the expectation that all of the markdown in Now that the markdown machinery is in the docbook generator, could we merge all three generators into one? |
|
@sandydoo working on it. ultimately all formatters for option docs should go into |
|
@pennae, awesome! What's the plan for the markdown renderer? Are we writing our own or should we try integrating MDRenderer from mdformat? |
|
@sandydoo somewhat inclined to write out own actually. mdformat looks like it'd make a good starting point, but nixpkgs markdown has a bunch of extensions that need to be converted or stripped in various ways. mdformat also does things we don't need (like line wrapping) and for asciidoc we'll need to do much of the indentation planning anyway, so the benefit of using an outside library may be negligible. haven't looked yet so can't say for certain |
Description of changes
don't use xsltproc to convert options.json into docbook xml. this also lets us skip using nix to convert json to xml, and using python to sort said XML for xsltproc. this one change saves about 10 seconds in the docs build on our machine and is thus probably good enough to include before the MD rendering code is generalized and stuck into a new tool.
also remove the olinkdb because it's not used at all and rather costly to build (taking 10 seconds on our system).
rendered output is largely unchanged except for a single option that used hard tabs for indentation. that one has more spaces now.
Things done
sandbox = trueset innix.conf? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)nixos/doc/manual/md-to-db.shto update generated release notes