Task Summary
A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. This step adds the first one: given a plan, the compiling service returns a single Python file that reads the same sources, applies the same operators in the same order, and prints its results.
An operator says how it reads outside the engine by implementing a StandaloneCodeGenerator trait, returning a block of pandas that names its inputs and outputs by position. The translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. A variadic port takes the whole list of upstreams rather than a fixed count, since any count an operator states would be wrong for some workflow. An operator that has no generator yet leaves a commented placeholder rather than a line that looks like it works, so the export is useful before every operator implements the trait.
Five operators implement it here to show the shape and to give the translator something real to walk: Distinct, Filter, Limit, Projection and Union. The rest of the operator set follows a family at a time.
First of the twenty-one steps listed in #8325. Everything else in the set that touches Scala needs the trait this one defines.
Task Summary
A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. This step adds the first one: given a plan, the compiling service returns a single Python file that reads the same sources, applies the same operators in the same order, and prints its results.
An operator says how it reads outside the engine by implementing a
StandaloneCodeGeneratortrait, returning a block of pandas that names its inputs and outputs by position. The translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. A variadic port takes the whole list of upstreams rather than a fixed count, since any count an operator states would be wrong for some workflow. An operator that has no generator yet leaves a commented placeholder rather than a line that looks like it works, so the export is useful before every operator implements the trait.Five operators implement it here to show the shape and to give the translator something real to walk: Distinct, Filter, Limit, Projection and Union. The rest of the operator set follows a family at a time.
First of the twenty-one steps listed in #8325. Everything else in the set that touches Scala needs the trait this one defines.