Support for variables wrapped around RDD

[ghislainfourny]

In the medium to remote future, we will want to smartly bind a variable with, instead of a materialized sequence of items, an "RDD wrapper" acting as a proxy to Spark in local expressions. This requires adapting the code of the dynamic context.

Example:

let $a := json-text("hdfs://.../file.json")
let $b := json-text("hdfs://.../file2.json")
return { a: count($a), b: count($b) }

The above FLWOR expression is local (i.e., the let clauses are executed locally, but wrapping on the RDD returned by json-text as if it were a local value in a blackbox), so that a prerequisite will be that local FLWORs are supported.

Note that this feature will be incompatible with FLWORs running on Spark, i.e., only "materialized" dynamic contexts can be used as RDDs because Spark forbids nesting.

[ghislainfourny]

This is being addressed.

[CanBerker]

I have just debugged the following nearly-identical query (file locations are updated and json-file function is used).
let $a := json-file("./src/main/resources/queries/conf-ex.json")
let $b := json-file("./src/main/resources/queries/conf-ex.json")
return { a: count($a), b: count($b) }

In the current state on master, this query is evaluated completely locally.

Is there anything else missing for completely addressing this issue ?

[ghislainfourny]

@CanBerker what is the status: are the two counts now evaluated in parallel? Thanks!

[CanBerker]

The current result:

rumble$ let $a := json-file("./src/main/resources/queries/conf-ex.json")
>>> let $b := json-file("./src/main/resources/queries/conf-ex.json")
>>> return { a: count($a), b: count($b) }
>>>
>>>
{ "a" : 5, "b" : 5 }
The query took 2289 milliseconds to execute.

I remember testing at the time and verifying the local API was used to evaluate this query as let clause is capable of storing RDDs as variables. If I'm not missing something I think that this issue is addressed.

[ghislainfourny]

Thanks! :)