Convert t2t to markdown, then to HTML

[seanbudd]

Link to issue number:

Closes #8734
Part of #15014
Related nvaccess/nvda-misc-deps#30, #16002, #15950, #15939, #15981

Summary of the issue:

In order to migrate to Crowdin, we must convert txt2tags to markdown.
In order transition safely, beta will build docs from t2t to markdown to html.
Eventually the t2t will be removed and the markdown will become the source of truth.

Description of user facing changes

The user Guide no longer has numbered sections

Translators and documentation writers now use extended markdown syntax rather than txt2tags.

Description of development approach

The build system performs certain pre-processing and post-processing when converting t2t to HTML.
This equivalent system should be retained - i.e. the translator and documentation contribution experience should remain the same for markdown to HTML.
Additionally, when converting t2t to markdown, new processing rules had to be created.

There is no universal standard for custom anchors in markdown.
As such text2tags doesn't have default rules for this. To retain our custom anchors, I added rules for a common markdown extended syntax.
Similarly a shortcut is used for generating table of contents.
See nvaccess/nvda-misc-deps#30

Setting the language code is done using the user_docs folder name, and the direction of RTL is manually set for the 3 current languages that require it, Persian, Arabic, Hebrew.

Special Catalan processing for adding hreflang attributes is converted to a markdown extension syntax of {hreflang=en}

The key commands file is generated using a custom made markdown python extension.

Testing strategy:

Tested running scons user_docs and checking the resulting files.

Known issues with pull request:

None

Code Review Checklist:

• Documentation:
  • Change log entry
  • User Documentation
  • Developer / Technical Documentation
  • Context sensitive help for GUI changes
• Testing:
  • Unit tests
  • System (end to end) tests
  • Manual testing
• UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers
  • Localization in other languages / culture than English
• API is compatible with existing add-ons.
• Security precautions taken.

[seanbudd]

Thanks @lukaszgo1 all feedback should now be resolved

[seanbudd]

@hwf1324

Yes, I have another proposal, maybe we can add linting and formatting tools.
For cases where file translation is used.

I would suggest opening up a separate issue.
PyMarkdown looks promising: https://github.com/jackdewinter/pymarkdown
It's far from complete and doesn't support diffs, but it could work as a tool for manually checking rather than CI/CD automated checking and failing builds.

[lukaszgo1]

No, I did not build from this branch before. The procedure you've proposed didn't help, as there were no .md files to remove. This can be reproduced as follows:

• Perform a clean checkout of the repository
• On master execute scons user_docs
• Switch to convertT2tToMarkdown ensuring to update sub modules
• Execute scons user_docs again

As long as the documentation was build before this is going to fail. Removing keyCommands.t2t from each language folders seems to be a work around. If this will prove difficult to fix I'd suggest at least adding a note to the known issues of this PR, as developers will certainly encounter this.
Also even when documentation builds successfully many .t2t files are marked as modified in git. Again if this is difficult to fix a note in the known issues should be added, though obviously a proper fix will be nicer.

[seanbudd]

Ah I see the problem. I have removed keyCommands.t2t from the .gitignore, along with build.t2tconf. This is to make them more visible to developers to ensure they remove them. During the transition period developers will have to perform git clean -f user_docs/*/keyCommands.t2t. There seems to be no easy way to do this and the transition period should be relatively short. This build error will only occur while we have t2t to md running.

The white space diffs in .t2t files could be avoided - however I plan to just commit these whitespace changes to beta once this PR is merged so that they are not annoying to devs.

[zstanecic]

will this be merged soon? I am just curious as a translator

[seanbudd]

Merging this alone won't affect translators - this is just a step towards getting the documents ready for translators