Reimplement jinja[spacing] to avoid use of regex

[ssbarnea]

This change rewrites jinja2 rule to make it use jinaja lexer and
avoid use of regexes for parsing, as they cause too many false
positives.

Fixes: #2301
Fixes: #2285
Fixes: #2241
Fixes: #2209
Fixes: #2208
Fixes: #2120

[cognifloyd]

fascinating use of the jinja lexer.
In the lexer-related work I've been doing, I've been using TOKEN_* constants from jinja:
https://github.com/pallets/jinja/blob/main/src/jinja2/lexer.py#L61-L109

Typically I do from jinja2 import lexer as j2tokens and then use j2tokens.TOKEN_OPERATOR or similar. What do you think of that idiom?

[ssbarnea]

@drybjed I am not sure how well you will take this but this change seems to uncover at least one genuine jinja2 syntax error in debops repository, but you might find yourself bit overwhelmed by the amount of feedback linter is providing:

Take it easy! And to quote from thread,... I hope you get paid by the hour... ;)

[ssbarnea]

@cognifloyd There are 4 reformatting test that are failing which I do not know how to fix. If you can have a look it would be great. That PR took all my weekend and I might really benefit from another pair of eyes to deal with these last ones.

All these tests were results of different errors I found while testing formatting of jinja2 expression on all eco repositories, so we should be pleased about the results.

Few minutes ago I got another idea,... what if I would try to use black itself to format the snippets, assuming that it might be used programately. Still, doing that would not be a joy as I would have to identify blocks of tokens and recombine them before I feed them to black,.. so maybe not a very good idea.

[cognifloyd]

Ew. Yes those are difficult. Here are some ideas to hopefully give you an idea of what to look at next.

[drybjed]

@ssbarnea Thanks for the heads up. I'll test the proposed changes and make fixes ASAP.

[drybjed]

@drybjed I am not sure how well you will take this but this change seems to uncover at least one genuine jinja2 syntax error in debops repository, but you might find yourself bit overwhelmed by the amount of feedback linter is providing:

Take it easy! And to quote from thread,... I hope you get paid by the hour... ;)

@ssbarnea I'm currently working on fixing all these warnings on ansible-lint v6.5.0 so that DebOps can be used to fix false-positives in v6.5.1. I wish for one thing - separate Jinja identation into its own rule so that it could be ignored if desired. There are small snippets of Jinja where I had to drop idented if/endif statements, and that's alright with me, but for larger pieces of Jinja, for example more extensive templates, please please please let me keep identation - it makes things much more readable.

[drybjed]

@ssbarnea All issues in DebOps have been fixed in the master branch. There are some false-positives though on the latest ansible-lint release.

[ssbarnea]

@drybjed Thanks. I got some progress today on them but again failed to pass all tests. That jinja spacing is a bit of a can of worms.

I already introduce a condition to skip the spacing checks when newlines are found inside the jinja blocks, basically because we assume user added them for indentation purposes. The scope of checking spaces was more towards simple blocks.