[0.1.3.1] Compilation error with filter on deeply projected variable - init_0_0

[chris-twiner]

Three runners deep filtering on results in a Databricks notebook fails. Selecting that field directly works, as does running expressions on it in the select (e.g. .isNull / is null), however when filter is used a compilation error is generated with a call to init_0_0 which does not exist.

Either:

1. bug / optimisation on deeply nested projections and three runners deep (e.g. nested / separated classes)
2. bug in quality expecting specific code gen approach (somehow incompatible with initMutableStates - which creates the init function - / the projection )
3. bug in databricks / oss spark

In order for this to occur in the code there would need to be a freshname provided by initMutableStates / splitExpressions which is then not accessible to the code calling it, which doesn't make sense considering it's only used at the creation site.

Further of interest is using .cache before the error code generating filter removes the issue (e.g. probably changing the projection approach).

As such repo needed and OSS run to see what code is actually produced, if it doesn't occur on OSS it's a likely DBR bug, if it does there may be options for workarounds.

This would need to be fixed / remediated on a 0.1.3.2 version or a new 0.1.3.1 site update to explain this (irrespective of OSS or Databricks).

[chris-twiner]

Test case fails against 0.1.3.1 on 14.3 and 17.3, it does not fail on 0.2.0 OSS 4.0/4.1.

[chris-twiner]

Using:

ruleEngineRunner(ruleSuite3, schema3, compileEvals = false, forceTriggerEval = false)

works in 0.1.3.1, conversely using 0.2.0 with the 0.1.3.1 true and true fails on all Databricks versions AND does on OSS 4.0/4.1.

So not DBR related.

[chris-twiner]

The issue is due to calling compile within an existing compilation and due to the result of a runner being used in the next runner.