Settle on one canonical TOC Tree implementation

[Mpdreamz]

Relates: #32

The {toctree} directive is not a Myst native directive it comes from Sphinx.

Currently this repository includes the sample documentation from https://github.com/elastic/markitpy-samples/tree/main/docs/source which were heavily geared to Sphinx.

Pending discussion #38 we'll most likely don't want to support {toctree} but instead use the Myst native way through myst.yml

See: https://mystmd.org/guide/table-of-contents

This has several advantages:

• Building the navigation does not require us parsing the whole file for a{toctree}
  • We don't have to enforce a {toctree} is always provided at the beginning/end of an index.md
• myst.yml allows grouping of related files in the same folder through children without them needing to be in a folder

Disadvantages

• Myst only uses a single toc myst.yml file so it's a one way door in the sense either everything needs to be in the TOC or nothing.
• It will flatten urls of nested files: https://mystmd.org/guide/table-of-contents#nested-files-will-have-flattened-urls
  • Although this has been addressed very recently: Support folder hierarchies and nested files in URLs without flattening them jupyter-book/mystmd#670

Proposal

Since myst.toc is very much geared towards smaller documentation sites or single Jupiter notebook html export I propose we follow its TOC in spirt:

• Allow a single toc definition control any nested children
• Allow nested toc definitions to isolate if needed
• Not a one way door, we can still include folder's without having to explicitly specify all its children
• Subfolders can define their own toc configuration
  • Unless a parent folder's toc defines its content explicitly this will result in a build error.

Example configuration

exclude:
  - notes.md
  - '**/ignore.md'
toc: 
  - file: index.md
  - file: config.md
  - file: search.md
     children:
       - file: search-part2.md
       - folder: search  
  - folder: my-folder1
    exclude:
       - '_*.md'
  - folder: my-folder2
    children: 
       - file: subpath/file.md
       - file: file.md
       - pattern: *.md
       - folder: sub/folder

Going over the bits in isolation:

exclude:
  - notes.md
  - '**/ignore.md'

Globally ignores any files matching the configured glob patterns.

toc: 
  - file: index.md
  - file: config.md

Ordered list so index.md and config.md appear first.

  - file: search.md
     children:
       - file: search-part2.md
       - folder: search  

Creates a collapsable section in the navigation with search.md being the root and search-part2.md and the contents of the search folder as its children.

  - folder: my-folder1
    exclude:
       - '_*.md'

Auto include my-folder1 if that folder contains a toc it will use that otherwise use the default rules for folder inclusion.

  - folder: my-folder2
    children: 
       - file: subpath/file.md
       - file: file.md
       - pattern: *.md
       - folder: sub/folder

An inline toc declaration for my-folder2. If that my-folder2 holds its own toc file too it will result in a build error.

The inline declaration automatically prefixes my-folder2 to children so that it can be renamed with ease.

future expansions ideas

We can expand this later with more options e.g:

  - folder: new-feature
     aliases: ['featureA', 'featureB']

To emit the folder as new-feature/* and featureA/* and featureB/*` as a means to safely rename a folder while links and rewrite rules get updated.

[bmorelli25]

Love this! Two things:

🗒 One of the contribution challenges with Docsmobile is the fact that the slug is independent of the file name. This means there's no obvious mapping between the source file structure and the URL path. This adds complexity to both finding source files, and to linking between pages. Have you thought about how this navigation structure will influence the HTML path for the built docs?

🗒 There's an additional request for V3 that is important to writers and relates to TOC tree implementation: The ability to add optional TOC section headings. These section headings should not be selectable. They would be only used to group related content together. For example, consider the following example config file:

toc: 
  - file: overview.md
  - file: explore.md
  - section: "Elastic Observability"
    - file: start-developing.md
       children:
         - file: start-dev-part2.md
         - folder: start-dev
    - file: sample-projects.md
       children:
         ...
    - file: apis.md
       children:
         ...
  - section: "Elastic Security"
    ...

The config file above might generate a TOC that looks like this:

[KOTungseth]

Will using a single myst.yml for ToC management pose challenges in flexibility for our large, multi-repo Docs sets?

Modular documentation

A single ToC might limit writers when contributing to the sub-ToCs of their Docs areas. We could consider implementing a strategy where the single myst.yml references other ToC fragments, creating a modular approach while maintaining a single entry point. However, this is how we work today, and it gets confusing. We could explore removing the manual effort by using scripts or tools that automatically generate or update the main myst.yml based on changes made in our repos.

Version-specific ToC modifications

With only one ToC, handling version-specific documentation changes could become complex. This is all dependent on what comes out of the version consolidation project, but my main concerns are Cloud, Logstash plugins, and APM agents. If different versions of the documentation need slightly different navigation, we’d need to manage those variations within the single myst.yml, which could be prone to errors or difficult to maintain. However, we do a good job of maintaining this today.

[bmorelli25]

Notes from meeting:

• Everything in the TOC should point to a file.
• One TOC file per documentation set.
• Configuration file name: docset.yaml
• For URL structure -- no alias support or custom mapping yet

[Mpdreamz]

@KOTungseth @bmorelli25 I am nearly done implementing this.

See https://github.com/elastic/docs-builder/blob/main/docs/source/docset.yml for a "real world" example.

One outstanding question:

Right now the navigation node is powered by the markdown's title:

The {toctree} directive allowed you to override the text of the page in the navigation.

Currently all pages under semantic-search/index.html all have the same title e.g:

https://github.com/elastic/docs-builder/blob/main/docs/source/elastic/semantic-search/amazon-bedrock.md

Is this a feature I need to add to docket.yml and/or the page's yaml frontmatter?

[bmorelli25]

Awesome!

And good question. Yes, this feature is very important for SEO purposes. We want to be able to set keyword-rich titles in documents, and be able to override those titles with custom nav/toc names. I think setting this override in the docset.yml file makes the most sense.

[bmorelli25]

The more I think about it, I'm indifferent to how this is solved (frontmatter or docset.yml). Using frontmatter also has pros, like searching the repo for the TOC name and be lead directly the source file.

[Mpdreamz]

I have a preference for yaml front matter too.

Since it can also define the title all presentational properties will be in one place.

I fear the toc in the docset.yml will become too noisy if it'd be defined there.

Will finish this ticket in the morning!

[bmorelli25]

Sounds good!