New docs: initial system setup

[m-vdb]

⬆️ Relates to https://github.com/RasaHQ/growth/issues/1436

We want to do the initial setup for the docs system:

• setup Docusaurus in the Rasa repo
• convert RST docs to MDX and have a simple way to convert future documents

The new docs will sit inside docs/ of the rasa repository. How the docs are build for hosting is part of #6166

Known issues

• Image width is not respected. E.g. See images here are much wider than in rst docs

• The chat bubbles here do not translate well here (I had expected them to appear something like the ones here. However these pages will likely be removed in 2.0 docs.) tracked in separate issue

• Glossary styling is lost: http://localhost:3000/docs/glossary

• the sidebar disappears on orphan pages (e.g. the channel pages) (it's a known issue and the workaround is to add the pages to the sidebars :/) - THIS SHOULD BE FINE; WE JUST NEED TO ADD THE PAGES SOMEWHERE. Check in with CSE about the new docs structure, maybe we can just hang them somewhere in the sidebar tree for now. we should not have any pages without sidebar items anyways.

• The autogeneration happens after variables are replaced. We need to find a way to support global variables. global variables can be used in JSX, but we need to find & replace original values with variable syntax manually Feature: referencing variables from markdown pages (placeholder substitution) facebook/docusaurus#395 (comment)

List of Sphinx directives we use

(we might want to sanity check these in a separate ticket)

• .. admonition::
• .. button::
• .. code-block::
• .. code::
• .. contents:: (obsolete, toc is in sidebar now)
• .. copyable::
• .. edit-link::
• .. glossary::
• .. group-tab::
• .. image::
• .. note::
• .. parsed-literal:: (only needed to parse variables, fixed in 0137605)
• .. raw::
• .. sourcecode::
• .. tab::
• .. tabs::
• .. toctree:: (obsolete, toc is in sidebar now)
• .. warning::
• .. container:: (can be done with remark-collapse)
• .. option:: (simple text formatting to figure out)
• .. pull-quote:: (one occurrence only, we can do it manually, and could use the standard blockquote >)
• .. testcode:: + .. testsetup:: (not sure we want to support these?)

Things we need to fix AFTER the initial converted docs are merged:

• .. program-output:: (separate ticket New docs: add support for program-output directive #6202)
• .. literalinclude:: (separate ticket New docs: include source files #6199)
• .. autoclass:: + .. automethod:: (separate ticket New docs: Autoclass and automethod generation from code #6194)
• .. include:: (only used for changelog, tackled in New docs: changelog generation #6170)
• .. juniper:: + .. runnable:: (obsolete, handled by new prototyper New docs: interactive code cells #6169)
• .. chat-bubble:: + .. conversations:: (tackled in https://github.com/RasaHQ/growth/issues/1501)

Let's create separate issues for the remaining open ones after the PR has been merged.

[tmbo]

let's collect all the items on this issue that we need to fix before we can replace the existing rst docs in rasa open source with their markdown version. I'll copy the list from the exploration PR

[tmbo]

I think .. autoclass: and .. automethod:: can be skipped. we need to generate the docs automatically based on our code anyways (so no need to support them in the rst -> md conversion)

[tmbo]

@m-vdb anything I should be working on here atm? Looks like you are going through the elements right now, can you add any todos we need to handle before replacing the rest docs on master with md as well as things that we can do afterwards?

[tmbo]

I don't think we need toctree nor edit-link for now. The toctree one is obsolete, because we have a table of contents on every page now. The edit link one is already present by default on every page. I think we should create a separate issue though if we want to make it more prominent in our styling of the docs (currently it is at the bottom of the page).

[tmbo]

juniper should be obsolete with the onboarding changes rick has been working on, right?

[m-vdb]

1. autodoc: yes, separate issue
2. i'm halfway through, will finish today, list the TODOs so that we can organize
3. toctree / edit-link: fine by me 👍

[m-vdb]

@tmbo I still think we should tackle literalinclude in this ticket. It should be easy enough to do with remark-sources and since there are a lots of occurrences, it'll be easier to do automatically. WDYT?

I updated the desc above. Some things to validate:

• can we drop testcode / testsetup? (and keep them as simple codeblocks)
• I'd open separate tickets for parsed-literal and program-output
• we have tickets for everything else

I'm starting with the small remaining elements now

[m-vdb]

🤔 OK I think I understand why you wanted to do that in a separate ticket. Basically, remark-sources won't be enough, because in some cases, we include a subset of the file. But we can definitely fork it and adapt to our needs. Will do in a separate ticket then 👍

[m-vdb]

currently trying to use remark-variables for variables substitution, without success

[tmbo]

Instead of testcode we can just use a code block for now. we can add automated tests for code snippets later. (we can drop testsetup completely, it is just meta info for the automated testing)

[tmbo]

whats the issue with the substituation, does it just not work or is there an error?

[m-vdb]

I think I found a solution. It simply doesn't work out of the box. I've tried remark-variables too. I found a solution to compute the variables before build of docusaurus and will commit it this afternoon 👌

[m-vdb]

@tmbo here is my solution to support variables: 0137605

WDYT?