Fix docs syntax highlighting errors

[felixfontein]

SUMMARY

This PR contains a couple of fixes of syntax highlighting errors reported in #50040. In most cases, the wrong lexer was used. Besides that, small problems with the Ansible output, YAML and Django lexers had to be fixed. I'll create upstream PRs for Pygments next.

Fixes #50040

CC @dagwieers @gundalow

ISSUE TYPE

• Bugfix Pull Request
• Docs Pull Request

COMPONENT NAME

docs

[ansibot]

cc @acozine
click here for bot help

[felixfontein]

Pygments Django PR: https://bitbucket.org/birkenfeld/pygments-main/pull-requests/796/django-lexer-add-support-for-and-operators/diff
Pygments YAML PR: https://bitbucket.org/birkenfeld/pygments-main/pull-requests/797/improve-yaml-mapping-lexing-fix-problems/diff

[felixfontein]

Both Pygments PRs have been merged for the next release, i.e. once that happens, we can throw out some of the lexers.

[gundalow]

Once Pygments has been a released do we remove the file docs/docsite/_extensions/pygments_lexer.py

[dagwieers]

Personally, I think we need to evolve to an Ansible lexer that:

• can distinguish between text-values and Jinja2 expressions (ie. when/until/...)
• includes yes/no booleans

That said, I am quite happy with the progress we have made. Priority should be on easy selection of docsite version. It is the number one inconvenience for our users. It should not be that hard to do IMO.

[dagwieers]

LGTM

[felixfontein]

@gundalow we can't remove it, since it contains the Ansible output lexer which isn't part of Pygments (yet). (For adding it to Pygments, I guess it needs to be more complete and better tested. Right now it's mostly good enough to run the output examples we have in the docs.)

[felixfontein]

@dagwieers I didn't notice that it doesn't know about all the other boolean values. That's something easy to add, though! (I guess we should add all of them, for completeness, and try to get it into Pygments.)

About when/until/...: that's a lot more tricky (and requires a huge redesign of the lexer). It would be nice to have that eventually, though.

[dagwieers]

@felixfontein I think the YAML lexer implements strict YAML 1.2, but we decided to allow both yes/no and true/false in documentation and examples. Adding all boolean values (i.e. on/off) is fine as well, since that's what Ansible does support anyway, but I doubt the strict YAML 1.2 is something they want to see changed.

[acozine]

@felixfontein thanks for doing this work - it looks great. As a next step, I think we need to document which lexers are available and when to use them. But meanwhile I'm going to merge this fix.

[felixfontein]

@acozine The plugin adds lexers yaml, jinja (alias django), yaml+jinja, and ansible-output. The first three are already provided by Pygments itself, but we've overridden them with some improvements. Where should we document them?

[acozine]

@felixfontein I'm thinking we could add them to the style guide, under https://docs.ansible.com/ansible/devel/dev_guide/style_guide/index.html#restructuredtext-guidelines

[webknjaz]

@felixfontein does pygments_lexer need to stay vendored?

[felixfontein]

@webknjaz parts of it definitely, for example AnsibleOutputLexer isn't in pygments (and there is no PR for it - I guess it should be more complete for that first). I'd have to check the rest, I don't remember which parts have already been integrated in pygments resp. whether something is missing.

[webknjaz]

@felixfontein

Sweet! Do you think it'd be possible to only keep the customized parts and just depend on the rest maintained somewhere else and installed by Pip?

[felixfontein]

AnsibleYamlLexerContext/AnsibleYamlLexer are equal to YamlLexerContext/YamlLexer in the current Pygments version, and AnsibleDjangoLexer is equal to DjangoLexer, and AnsibleYamlJinjaLexer is equal to YamlJinjaLexer. So yeah, we can get rid of all of them. I can create a PR for that.

Should I add a new enough pygments version to https://github.com/ansible/ansible/blob/devel/docs/docsite/requirements.txt ? Or where should I add that requirement?

[felixfontein]

(FYI: the required patches so that the code is actually equivalent are in Pygments 2.4.0 and newer.)

[webknjaz]

@felixfontein there's also a constraints file: https://github.com/ansible/ansible/blob/devel/test/runner/requirements/constraints.txt