Add experimental command to write generated curation to documentation extension files

[d-ronnqvist]

Bug/issue #, if applicable: rdar://98528457

Summary

This adds an experimental command to write the auto-generated curation to documentation extensions files.

In some workflows outside of build integrations this can be a way to generate a starting point for curation and to determine what symbol disambiguation is needed to curate symbols with name collisions.

The new command also has two options to control which symbols to emit documentation extensions files for: --from-symbol <symbol-link> and --depth <limit>

OVERVIEW: Write documentation extension files into the DocC Catalog with one that supports a static hosting environment.

USAGE: docc process-catalog emit-generated-curation [<documentation-catalog>] [--additional-symbol-graph-dir <additional-symbol-graph-dir>] [--output-path <output-path>] [--from-symbol <symbol-link>] [--depth <limit>]

INPUTS & OUTPUTS:
  <documentation-catalog> Path to the DocC Catalog ('.docc') directory.
  --additional-symbol-graph-dir <additional-symbol-graph-dir>
                          Path to a directory of additional symbol graph files.
  --output-path <output-path>
                          The location where docc writes the transformed catalog.
        If no output-path is provided, docc will perform an in-place transformation of the provided DocC Catalog.

GENERATION OPTIONS:
  --from-symbol <symbol-link>
                          A link to a symbol to start generating documentation extension files from.
        If no symbol-link is provided, docc will generate documentation extension files starting from the module.
  --depth <limit>         A depth limit for which pages to generate documentation extension files for.
        If no depth is provided, docc will generate documentation extension files for all pages from the starting point.
        If 0 is provided, docc will generate documentation extension files for only the starting page.
        If a positive number is provided, docc will generate documentation extension files for the starting page and its descendants up to that depth limit (inclusive).

OPTIONS:
  -h, --help              Show help information.

For example, passing --depth 0 will only emit a documentation extension file for the module (emitting the auto-generated curation for top-level symbols).

swift run docc process-catalog emit-generated-curation \
  YourProject.docc \
  --additional-symbol-graph-dir /path/to/symbol-graph-directory \
  --depth 0

Passing --from-symbols SomeClass will emit documentation extension files for "SomeClass" and its sub hierarchy.

swift run docc process-catalog emit-generated-curation \
  YourProject.docc \
  --additional-symbol-graph-dir /path/to/symbol-graph-directory \
  --from-symbol "SomeClass"

Passing --from-symbols SomeClass and --depth 1 will emit documentation extension files for "SomeClass" and its direct descendants.

swift run docc process-catalog emit-generated-curation \
  YourProject.docc \
  --additional-symbol-graph-dir /path/to/symbol-graph-directory \
  --from-symbol "SomeClass" \
  --depth 0

Passing neither --from-symbols nor --depth will emit documentation extension files for every symbol in the project.

swift run docc process-catalog emit-generated-curation \
  YourProject.docc \
  --additional-symbol-graph-dir /path/to/symbol-graph-directory

If you want the generated documentation extension files to be emitted in a separate location you can pass an explicit output path.

swift run docc process-catalog emit-generated-curation \
  YourProject.docc \
  --additional-symbol-graph-dir /path/to/symbol-graph-directory \
  --output-path NewCatalog.docc

Dependencies

None

Testing

In a "process catalog" workflow

This functionality is mainly intended for workflows that aren't tied to a build integration and that can call a docc command other than "convert." Because of this, most of the configurability is only available in the new process-catalog command

• Emit generated curation for symbol graphs;

   swift run docc \
   process-catalog \
   emit-generated-curation \
   --additional-symbol-graph-dir /path/to/symbol-graph-directory
  

  A new "Generated.docc" catalog will be created in the current directory

• Inspect the generated documentation extensions; find Generated.docc

• Manually recreate some of the generated curation.

  • Open one of the generated documentation extension
  • Create a few new topic sections and move some of the generated curation links into those topic sections
  • Remove the remaining generated curation links
  • Remove the other generated documentation extension files
  • Rename the generated DocC catalog

• Emit generated curation in addition to what's manually curated

   swift run docc \
   process-catalog \
   emit-generated-curation \
   YourRenamedCatalog.docc \
   --additional-symbol-graph-dir /path/to/symbol-graph-directory
  

• Inspect the generated documentation extensions; find YourRenamedCatalog.docc

  • The removed documentation extensions files will be regenerated
  • Open the documentation extension file with manual curation
    • Symbols that are manually curated doesn't get generated curation links

• Remove some manual curation and some generated documentation extension files again

• Add an article to the documentation catalog

  echo "# An article\n\nThis is an article" > YourRenamedCatalog.docc/article.md
  

• Emit generated curation into a different output location

   swift run docc \
   process-catalog \
   emit-generated-curation 
   YourRenamedCatalog.docc \
   --additional-symbol-graph-dir /path/to/symbol-graph-directory \
   --output-path NewCatalog.docc
  

• Inspect the input documentation catalog; find YourRenamedCatalog.docc
  This catalog shouldn't be modified by the previous command

• Inspect the output documentation catalog; find NewCatalog.docc

  • The generated curation links and generated documentation extension files should exist in the catalog
  • The article and other non-generated files are not copied over from the input catalog to the output catalog.

• Emit only a documentation file for the module

   swift run docc \
   process-catalog \
   emit-generated-curation 
   YourRenamedCatalog.docc \
   --additional-symbol-graph-dir /path/to/symbol-graph-directory \
   --depth 0 
  

• Inspect the documentation catalog; find YourRenamedCatalog.docc

  • Only one generated documentation extension file should exist in the catalog (for the module)

• Emit documentation files for the specific symbol sub hierarchy

   swift run docc \
   process-catalog \
   emit-generated-curation 
   YourRenamedCatalog.docc \
   --additional-symbol-graph-dir /path/to/symbol-graph-directory \
   --from-symbol SomeClass
  

• Inspect the documentation catalog; find YourRenamedCatalog.docc

  • Only generated documentation extension files for "SomeClass" and its descendants should have been added.

• Try emitting documentation files for the symbol that doesn't exist

   swift run docc \
   process-catalog \
   emit-generated-curation 
   YourRenamedCatalog.docc \
   --additional-symbol-graph-dir /path/to/symbol-graph-directory \
   --from-symbol SomeClas
  

  • There should be a warning that "SomeClas" wasn't found. If the symbol path is similar to a real symbol there may be suggestions on how to fix the symbol link.

In a "convert" workflow

Add the --experimental-modify-catalog-with-generated-curation flag to a docc convert call that uses a documentation catalog,

After the conversion, the catalog may contain generated documentation extension files with auto-generated curation.

• Create an empty catalog; mkdir Empty.docc
• Convert some symbol graphs;

  swift run docc convert \
  Empty.docc \
  --experimental-modify-catalog-with-generated-curation \
  --additional-symbol-graph-dir /path/to/symbol-graph-directory
  

• Check the content of the catalog; find Empty.docc

Checklist

Make sure you check off the following items. If they cannot be completed, provide a reason.

• Added tests
• Ran the ./bin/test script and it succeeded
• Updated documentation if necessary

[heckj]

❤️
FWIW - I love the idea. I'm currently working around this by generating documenation and using jq to go through and pull all the references from LinkedEntities to get an exhaustive list, in turn dumping that into a text file in symbol format to do manual curation.

[d-ronnqvist]

Yes, without IDE integration there isn't a great way to get to this information today. I'm not sure if this is the best way to surface this information but I think it's a decent start that we can evolve as more people use it and provide feedback on it.

Inspecting the linkable entities is a decent workaround but it's has some drawbacks since the path to a page has more restrictions than the link to a page. For example, <(_:_:) and >(_:_:) are valid links but they're not valid paths, so the <, and > characters are replaced with _ in the path. This makes both paths _(_:_:) which introduce the need for disambiguation. Since both symbols are operators, the only other way to distinguish them is by adding the hash of their unique identifiers. This means that the path you'll find in the linkable entities would be something like _(_:_:)-abc123 and _(_:_:)-def456 which aren't the best links to those pages.

[d-ronnqvist]

@swift-ci please test

[d-ronnqvist]

@swift-ci please test

[franklinsch]

This is incredibly useful, thank you for adding this! I've been using this command and one thing I've noticed is that if symbols don't have children, it doesn't emit a documentation extension, which given the name emit-generation-curation, is correct behaviour since there's no curation to generate for pages with no children.

Do you see value in bringing support for generating documentation extensions for symbols that don't have children though? A command such as docc process-catalog generate-documentation-extension --symbol "Foo/Bar" could work well for this. This isn't the focus of your PR though, so it would make sense to address this as part of future work, unless you think the feature you're contributing here is incompatible with the command I'm suggesting above (which I don't think it is).

In the meantime, I wonder if we should make the help text of emit-generated-curation clearer. It's currently:

Write documentation extension files into the DocC Catalog.

but there could benefit in clarifying that the purpose of this is generating curation, and hence symbols with no children don't get an extension generated.

[d-ronnqvist]

@swift-ci please test

[d-ronnqvist]

This is incredibly useful, thank you for adding this! I've been using this command and one thing I've noticed is that if symbols don't have children, it doesn't emit a documentation extension, which given the name emit-generation-curation, is correct behaviour since there's no curation to generate for pages with no children.

Yes, the intended behavior is that DocC will only write a file if there's any autogenerated curation in it.

Do you see value in bringing support for generating documentation extensions for symbols that don't have children though? A command such as docc process-catalog generate-documentation-extension --symbol "Foo/Bar" could work well for this. This isn't the focus of your PR though, so it would make sense to address this as part of future work, unless you think the feature you're contributing here is incompatible with the command I'm suggesting above (which I don't think it is).

We could add functionality like that in a separate command. I intentionally made this a subcommand of a new process-catalog command so that there would be a place to add new commands like this and a place to discover similar commands (docc process-catalog --help).

On its own I don't feel like a command to only generate documentation files is very useful. To do it for a specific symbol you already need to write the symbol link for that symbol and to generate large numbers of empty extension files is a bit of an uncommon use case. That said, it's probably very easy to implement if someone has a need for it and wants to contribute a catalog processing subcommand.

Maybe there are more specific use cases where DocC can automate more of the work? For example, overriding a symbol in a documentation extension where DocC could insert the directive to configure the override behavior and copy over the in-source documentation for that symbol as a starting point.

In the meantime, I wonder if we should make the help text of emit-generated-curation clearer. It's currently:

Write documentation extension files into the DocC Catalog.

but there could benefit in clarifying that the purpose of this is generating curation, and hence symbols with no children don't get an extension generated.

I updated this to "Write documentation extension files with markdown representations of DocC's automatic curation." and also added a brief discussion section with some information for people who are new to this command and/or new to curation in DocC:

Pass the same '<catalog-path>' and '--additional-symbol-graph-dir <symbol-graph-dir>' as you would for docc convert to emit documentation extension files for your project.

If you're getting started with arranging your symbols into topic groups you can pass '--depth 0' to only write topic sections for top-level symbols to a documentation extension file for your module.

If you want to arrange a specific sub-hierarchy of your project into topic groups you can pass '--from-symbol <symbol-link>' to only write documentation extension files for that symbol and its descendants. This can be combined with '--depth <limit>' to control how far to descend from the specified symbol.

For more information on arranging symbols into topic groups, see https://www.swift.org/documentation/docc/adding-structure-to-your-documentation-pages.

[daniel-grumberg]

Are we expecting this update?

[d-ronnqvist]

No. I'll fix it when I merge main.

[daniel-grumberg]

IIUC this only checks if the reference we are looking at was curated as a direct descendent of the node we are currently processing?

[d-ronnqvist]

Nice catch. Yes, that is what this is checking. However, I had already tested that this works with any curation so I went to look at why that worked... It turns out that AutomaticCuration.topics(for:withTraits:context:) already checks if the link is curated anywhere, so this check isn't needed.

[daniel-grumberg]

Fantastic! I think that will simplify the code somewhat.

[daniel-grumberg]

This is a slightly confusing variable name, took a little while to understand this measures the longest line without a comment. I wonder if generating the String components first and then calculating padding might be more readable, even though it introduces an extra loop over the links, something along these lines:

let linkTextFragments = links.map { (link: "\n- ``\($0.link)``", comment: $0.comment.map { " <!-- \($0) -->" }) }
let longestLinkLine = linkTextFragments.map(\.link.count).max()!
let curationText = linkTextFragments
    .map { $0.comment == nil ? $0.link : $0.link.padding(toLength: longestLinkLine, withPath: " ", startingAt: 0).append($0.comment!) }
    .joined()
text.append(curationText)

[d-ronnqvist]

I incorporated the code that wraps the link and comment text in markup into the compactMap above. That made the code that computes the longest link and appends the markup strings easier to follow.

[d-ronnqvist]

@swift-ci please test