Exploration of new docs build system

[tmbo]

Proposed changes:

• Exploration of new docs build system

• related to https://github.com/RasaHQ/growth/issues/1436

• not all pages have been added to the new system. all of them are converted, if you are missing a page you'd like to take a look at you need to add it to newdocs/sidebars.js

To view the page, navigate into newdocs, run yarn to install dependencies and then yarn start. A browser page with the docs should pop up.

The converter and necessary changes to fix some of the markdown issues can be found here RasaHQ/sphinx-markdown-builder#1 - should be extended and adapted to fix more issues.

How to build the MDX docs based on RST

All of the new docs pages have been created automatically. If there are any changes in the RST docs (e.g. by new features or changes merged into master), the MDX docs can be regenerated:

• checkout and install https://github.com/RasaHQ/sphinx-markdown-builder/pull/1 using git clone git://github.com/RasaHQ/sphinx-markdown-builder.git, git checkout convert-docs and pip install -e . (this should be in your rasa poetry env, e.g. try running poetry shell before the pip install)
• checkout this branch and cd docs and run make markdown

This will tell sphinx to build the docs and output markdown (instead of the usual HTML). The converter is pretty flexible and if we notice any issues, we can further customize the sphinx-markdown-builder based on the above PR.

While converting, sphinx will create somewhat of an AST and then walk through these nodes. for each node in the doc there is a hook that decides how to convert that node into markdown.

Remaining work before this can be merged is tracked in #6165

TODO

• move newdocs to docs. rename existing occurrences, e.g. https://github.com/RasaHQ/rasa/pull/6043/files#diff-793229551c14ebd7f4c0f9236d054ee0R40 0777558 and 4b18077
• remove FIXMEs in scripts/push_docs_to_branch.sh script 4b18077
• fix /docs/ links everywhere f758baf
• support top-level CHANGELOG in the docs (similar setup than variables.json) 968e4a1
• fix make docs (failing because of redoc)
• setup Netlify redirection (see https://github.com/RasaHQ/rasa-website/pull/763)
• links appear to be broken after last alpha update e298750
• prepare prototyper move Interactive code cells #6298

Fixes #6170

[indam23]

Trying to view it, getting an error that src/css/custom.css is missing

[tmbo]

@melindaloubser1 should be fixed after you pull - for some reason src/ was in git ignore... 🤷

[indam23]

Thanks!
So I don't see our docs, just this page:

[alwx]

Same. Cannot find any of Rasa pages.

[alwx]

@melindaloubser1 just try to open any of the pages manually. For example: http://localhost:3000/docs/changelog

[indam23]

ohh thanks!

[tmbo]

fixed it in a push should be good now 👍

[tmbo]

thanks for trying this out

[indam23]

Love the pinned TOC on the right!
(And the collapsible sidebar, obviously 👌)

[erohmensing]

• also love the pinned toc! (and code headers look nice in the toc)
• changelog header links have persistent ids (instead of indexed ones) so they won't change when we add new entries above 🙌🏻
• looking at their docs on versioning, they argue that you shouldnt create new docs for every patch since there shouldn't be new features (fair). Can we build the docs from 1.10.x live like we do master, so we don't have to cut patches for docs updates?
• i like the previous and next navigators at the bottom of the page
• would be nice if we could decide per-page if we want depth 1 or 2 on the pinned toc (e.g. changelog), not sure if that is possible

this looks very exciting and promising!

[tmbo]

my best guess right now is that in the beginning we'll probably not use multi-version and keep a single docs version. we can try to aim more for an approach that integrates changes made into the documentation, e.g. here is an example how python does it https://docs.python.org/3.8/library/functions.html#print:

Changed in version 3.3: Added the flush keyword argument.

[tmbo]

I've updated the newdocs with a new export, it looks pretty good now. All the things you see in the docs are automatically generated - so we can run this once we are ready even if the rst docs change in the meantime.

[tmbo]

added some more details to the PR description, explaining how the automatic translation from RST to markdown (MDX) works.

[amn41]

looks awesome!

[m-vdb]

looks amazing 🤗 just ran it locally and worked like a charm!

[m-vdb]

Keeping this one open until @erohmensing reports more issues with the auto conversion. We'll try to fix most of them in this PR.

[m-vdb]

@tmbo this is it. we can merge this once the build passes 🚀 I've written a short doc on the new setup to share with the team. Can you take a look here?

[tmbo]

awesome, very hyped about this 🚀

the page looks good, let's make sure to also include a slide in the squad sync tomorrow to make sure everyone is aware of the most important parts (e.g. docs are in md now) 👍

[m-vdb]

yes will do as soon as you share the slides ✅