Defer variables used in deferred nodes

[Joeoh]

As a follow on to the discussion in the PR here: #421

This PR proposes a solution for two problems: Handling variables used within a deferred block and handling variables used in deferred blocks that occur at lower scopes.

Handling variables used within a deferred block

As discussed in #421 we want a way to mark variables as deferred while we are processing so that any subsequent uses of the variable are not processed. This is achieved here by scanning the deferred node each time and finding variables used within it. These variables are then put back into the context as DeferredValues with their original value. Any subsequent uses of these variables will result in a DeferredValueException and be handled like any other deferred variable. Before adding DeferredValuesUtils this kind of logic only existed in internal HS code -I believe because it is a small bit hacky. Before now, anyone else making use of the deferred functionality would have had to have their own way to rebuild a Deferred Render Context for their 2nd stage render.

I have also added support for finding yet undefined variables used within deferred blocks and marking them as deferred for example

{% if deferred = 'foo' %} 
{% set newVariable = 'deferred was foo' %}
{% else %}
{% set newVariable = 'deferred was not foo' %}
{% endif %}
{{ newVariable }}

Previously this would have output newVariable as an empty undefined var. Now it will be put in the context as deferred and so will be re-output as {{ newVariable }}

handling variables used in deferred blocks that occur at lower scopes

I have observed issues in using the previous implementation of deferred variables in a HS setup where things are rendered in lower contexts (eg modules) and any deferred values are lost. These changes copy any deferred values to the parent scope, again with their original value.

I can think of some cases where this might go wrong. If you had a template, that imported the same module multiple times, the first instance of the module would muddle the global scope with its values. I don't think this more global scope should cause issues as any code making use of deferred values should be preserved and the order of execution should be consistent. I added handling for deferred values to SetTag to handle this case.

[boulter]

The tests look thorough, so I trust that this works as expected. I've got a few othr comments you might want to consider.

[boulter]

Could you use the name defined here?

jinjava/src/main/java/com/hubspot/jinjava/lib/tag/SetTag.java

Line 65 in 5cb2463

public static final String TAG_NAME = "set";

[boulter]

I don't think this will work with multivariable assignments like set myvar1, myvar2, myvar3, myvar4 = ... I'm not sure you'd want to though as that might make things very complex.

[Joeoh]

Nice catch. I added this regex to be more specific for looking for sets as we don't have the existing context to compare matches to filter out noise. I think I will change this (and the other usages) to filter out any reserved tokens instead.

[boulter]

Might be nicer to put these in fixture files like some of the other tests.

[boulter]

comment needed?