Recursive render fails on raw/endraw

[Nachtfeuer]

Following problem:

$ PYTHONPATH=$PWD scripts/spline --definition=examples/docker.yaml --tags=remove-docker-container
2018-08-16 05:50:04,315 - spline.application - Running with Python 2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]
2018-08-16 05:50:04,321 - spline.application - Running on platform Linux-4.9.0-6-amd64-x86_64-with-debian-9.4
2018-08-16 05:50:04,321 - spline.application - Current cpu count is 4
2018-08-16 05:50:04,321 - spline.application - Processing pipeline definition 'examples/docker.yaml'
2018-08-16 05:50:04,339 - spline.application - Schema validation for 'examples/docker.yaml' succeeded
2018-08-16 05:50:04,382 - spline.tools.version - Using tool 'Bash', Version(4.4.12)
2018-08-16 05:50:04,382 - spline.tools.version - Using tool 'Docker', Version(17.12.1)
2018-08-16 05:50:04,383 - spline.tools.version - Using tool 'Spline', Version(1.11)
2018-08-16 05:50:04,383 - spline.components.stage - Processing pipeline stage 'example'
2018-08-16 05:50:04,384 - spline.components.tasks - Processing group of tasks (parallel=no)
2018-08-16 05:50:04,386 - spline.components.tasks - Processing Bash code: start
2018-08-16 05:50:04,397 - spline.tools.filters - render(syntax error): unexpected '.'
2018-08-16 05:50:04,398 - spline.components.tasks - Pipeline has failed: leaving as soon as possible!

The yaml contains {% raw %} and {% endraw %} which seems to be rendered twice and second
time the remaining string contains {{.ID}} which cannot be resolved.

When I remove the recursive replace all is fine.

[Nachtfeuer]

@jbenden Could you please have a look at that?

[jbenden]

Sorry on the delay.

There are a few possibilities here, let's figure out which seems best.

1. Undo the recursive templating, making templates ugly. :)
2. Follow one of the suggestions in this thread: Using {% raw %} in templates does not stop jinja2 variable substitution ansible/ansible#4638 However, I did not add the #jinja2:settings... syntax, but could.
3. Find a way to escape raw well enough for this to work. I've tried two techniques that failed, so it could be problematic.

Thanks,
-Joe

[Nachtfeuer]

Hi Joe :)

Well 1.) for sure is not the option.

Thinking about 3. ...

Once a template contains (before rendering) the raw section we know
that the loop has to stop (after rendering) because the raw part has gone.

[jbenden]

Just after I say that, I've found a solution that is not too horrid. Sigh. PR will follow.

Thanks,
-Joe

[Nachtfeuer]

I'm currently at the point to say: "stop" We should rethink that implementation and we can do that by copying the render code into a local demo project and looking forward how to solve the problem in a way that is acceptable.

Following example:

model:
    level1: |
        {% raw %}echo "{{ .VARIABLE_DOES_ALSO_NOT_EXIST }}"{% endraw %}
        echo "hello world 1!"
    level2: |
        {% raw %}echo "{{ .AND_THIS_VARIABLE_DOES_ALSO_NOT_EXIST }}"{% endraw %}
        echo "hello world 2!"

pipeline:
    - stage(Example):
        - tasks(ordered):
            - shell:
                script: |
                    {% raw %}echo "{{ .VARIABLE_DOES_NOT_EXIST }}"{% endraw %}
                    echo "hello world 0!"
                    {{ model.level1 }}

Of course stopping by first occurence (as I proposed) won't work because then the raw statements on the next levels remain and then the Bash code does not now what to do with it. For me it looks like that we have to search for each occuerence of a {{ ... }} that is not part of a raw section and to render that part only and to replace the original {{ ... }} ... on each level ... recursive. That's not really trivial.

Using a sentinal don't look to me understandable .... I prefer a logic that can be read and well understood also it means that it's necessary to define a clever algorithm.

Your idea "recursice" is good (still to mention) but the implementation start to look questionable.
Let me finish my current tasks and I jump in ... no need to hurry now :)

[jbenden]

I agree on stop. I spent a lot of time on this, coming up with complex solutions; however, they all failed with raw nested to some degree.

I ended up with the current solution, which completely works. It is not the prettiest, but it is small and contained to this region of code. So while a tad bit hackish, I figured it is well enough contained and well enough tested; to be accepted as the solution for now.

A "real" / non-hackish solution would involve processing at the AST level; which seem far more brittle than the current solution.

Ultimately, I believe it should have been, or should be, implemented in Jinja2.

[Nachtfeuer]

removed in milestone 1.11 (for now) ... until we have a better solution.