Scaling verification of workflow to Python translation — looking for input

[kz930]

We've been working on translating Texera workflows into standalone Python scripts and verifying that the translation is correct. We've built a working prototype for making verification scale: a shared fixture plus automatic per-operator configuration. It already runs — operators are auto-discovered and each is verified against the shared fixture, and concrete cases like CSVScan and ChoroplethMap pass end-to-end. Before we invest more in widening coverage, we'd like input on whether this shared-fixture direction is the right foundation.

Background: the two workflows

Translation. A Texera workflow is converted into a standalone Python script. The UI sends the workflow to the translator, the translator asks each operator for its code fragment, and the fragments are stitched together into one .py file that is returned to the user.

Verification. To check the translation is correct, we run each operator two ways: first through the native Texera execution path (the ground truth), then through the generated Python script. We compare the outputs from both paths. If they match, the translation is considered faithful.

Problem: verification does not scale

The current verification approach requires one hand-written test case per operator, and each test case needs its own input data and operator configuration.

This does not scale. Texera has around 150 operators, and more are added over time. Manually writing and maintaining a fixture for every operator is too much work.

Options considered

1. Keep hand-writing a fixture per operator (status quo). Accurate, but the maintenance cost grows with every new operator — the thing we're trying to get away from.

2. One shared CSV fixture + automatic configuration. Instead of a separate fixture per operator, build a single large shared CSV and auto-configure each operator against it, so a new
operator needs little or no additional hand-written test code. (This is the direction we currently lean toward — details below.)

3. Per-operator generated/synthetic input. Generate input tailored to each operator's declared schema on the fly. More precise per operator, but closer in spirit to the status quo and
harder to keep meaningful across domains.

Our current position / proposed plan

For most table-based operators, use one large shared CSV fixture. Each operator is tested against this shared input table, and its configuration is filled automatically.

Designing the shared CSV. We scan operator fields and identify what each field needs — numeric columns, text columns, date columns, country codes, p-values in (0,1), price-related columns,
and other domain-specific values — and add meaningful columns to cover those needs.

Testing an operator. Feed the shared CSV into the operator, auto-fill its configuration, run both the native path and the generated Python path, and compare outputs. Since both paths use
the same input and the same configuration, any output difference should indicate a translation bug.

Choosing a column for a field (two steps):

1. Match by type — numeric fields get numeric columns, text fields get text columns, date fields get date columns, and so on.
2. Match by domain when type isn't enough — e.g. a field that expects a country code should get a country-code column, not just any text column.
   If a field can't be filled meaningfully, we flag it rather than silently passing the test.

Filling non-column settings (by type):

• Enum / mode fields → test each option.
• Boolean fields → test both true and false when appropriate.
• Numeric settings → use the operator's declared default, falling back to 1 if none.

Example: ChoroplethMap

ChoroplethMap draws a world map. It has two column-reference fields:

• locations — needs a country code
• color — needs a numeric column

Against the shared CSV:

id	name	score	iso_country
1	alice	0.5	USA
2	bob	1.0	CHN
3	carol	1.5	JPN

auto-config fills the two fields using the two steps above:

• color → id — match by type (first numeric column)
• locations → iso_country — match by domain (a plain text column like name would render nothing, so it needs the country-code column)

This gives the config { locations: "iso_country", color: "id" }. Both paths then run on the same input + same config. If the outputs match, the translation is faithful. locations = iso_country is
exactly the "match by domain" case: with any text column the map draws nothing and the test would be vacuous — the country-code column makes it a real test.

An example workflow ending in ChoroplethMap

Each box in the diagram becomes one code block below; each arrow — df1, df2, df3 — is the DataFrame variable handed to the next operator. The translator walks the workflow in topological order and stitches these fragments into one script.

Each operator emits its fragment using placeholders — it doesn't know its neighbors. For example, Filter always generates:

out1df = in1df[(in1df["score"] >= 1.0)].reset_index(drop=True)                                                                                                                                                 

The translator sees that Filter's input comes from CSV Scan (df1) and assigns Filter's own output df2, then substitutes the placeholders:

df2 = df1[(df1["score"] >= 1.0)].reset_index(drop=True)                                                                                                                                                        

Doing this for every operator in topological order — and concatenating the results — produces the full script below:

# [CSV File Scan]
df1 = pd.read_csv(filepath_or_buffer="choropleth_test.csv", sep=",", encoding="utf-8", header=0)

# [Filter]
df2 = df1[(df1["score"] >= 1.0)].reset_index(drop=True)

# [Projection]
df3 = df2[["score", "iso_country"]].copy()

# [Choropleth Map]
df3 = df3.dropna(subset=["iso_country", "score"])
if df3.empty:
    print("Choropleth map error: No valid rows left")
else:
    fig = px.choropleth(df3, locations="iso_country", color="score", color_continuous_scale=px.colors.sequential.Plasma)
    fig.update_layout(margin={"r":0,"t":0,"l":0,"b":0})
    fig.write_json("output.json")
    fig.write_html("output.html")
    print("Choropleth map saved to output.json and output.html")                                                                                                                                               

Special cases

• Image and ML/sklearn operators don't fit a plain table CSV — they need binary data, image data, or model/training-style inputs. These should use separate fixtures.
• Whole-workflow correctness. Every operator passing individually doesn't prove the stitched Python script is correct. So we'd also test a small set of complete workflows (native vs.
  generated). Only a few should be needed, since the stitching mainly has to cover a handful of shapes — simple chains, fan-out, joins, and multiple-output operators.

Next steps (tentative)

1. Create one shared CSV file for testing.
2. Write code that automatically fills in each operator's settings.
3. Add small annotations to operators where needed.

What we'd like input on

• Is a single shared CSV the right foundation, or does it fall apart for too many operators?
• Is automatic type/domain-based configuration robust enough to trust, or will it produce too many false passes?
• Are there better alternatives we haven't considered for making per-operator verification scale?

[kz930]

@Yicong-Huang Could you share any advice on this? Thank you so much!

[chenlica]

@kz930 Before other people provide comments, please use a real example with a diagram to explain the details.

@carloea2 : as you are the mentor of this project, please chime in to provide your thoughts.

[kz930]

Thanks for giving suggestions. I added a concrete example with a diagram to the post above — it walks one
operator through auto-config (type + domain matching) and the two-path comparison.

[carloea2]

In my opinion, for each operator, we should define a function that returns a small sample dataset. This dataset can then be used as input to test the outputs in both Python Translation and the Texera engine.

I also think it would be a good idea to include a method for instantiating an operator instance that can consume this input by filling in the required properties. These properties may belong to different domains, so automating this process reliably would be difficult, or even impossible.

[kz930]

Agreed — fully automating property-filling across domains isn't reliable, so we don't aim for that. In practice it splits in two: type covers most column fields automatically (from the operator's existing attributeTypeRules), and only value-domain fields (e.g. a p-value in (0,1), a country code) need a one-line annotation on the field — essentially your "operator declares how to fill itself," reduced to one line rather than a full config method. So most operators need no per-operator code.

On the per-operator sample dataset: we currently lean toward one shared table with typed/domain columns, so an operator just points at a column instead of shipping its own data. But your sampleData()-per-operator is a clean fallback for inputs the shared table can't express (image / model / training-style).

[chenlica]

@kz930 Can you draw a diagram to visualize a workflow and explain how it's translated? That could be easier to understand than your current diagrams.

[kz930]

Thanks! I've added a concrete example to the discussion — a small workflow (CSV Scan → Filter → Projection → ChoroplethMap) drawn as operator boxes, with the Python script it translates into right below.

[chenlica]

@kz930 Per our offline meeting, please describe what changes we want to make for each operator. We need to first finish existing operators, then make a plan for future operators.

[kz930]

@chenlica Following our meeting, here is the concrete per-operator change, using the two simple cases we discussed.

For each operator, making it verifiable should be a small and uniform change:

1. Implement generateStandaloneCode() and producesDataFrame() for the operator’s Python translation.
2. Let auto-config fill the operator fields using the shared fixture. Type-matching should cover most fields; when type alone is not enough, add a one-line @SampleColumn hint.
3. Once the operator implements generateStandaloneCode(), OperatorBehaviorSpec auto-discovers it and compares the native Texera output with the generated Python output in the correct bucket: table, Plotly JSON, or HTML. No per-operator test needs to be written.

Two simple examples:

CSVScan source/table operator:

• generateStandaloneCode() emits pd.read_csv(...).
• No annotation is needed.
• It is verified with assert_frame_equal between the native output and the generated Python output.

ChoroplethMap visualization operator:

• generateStandaloneCode() emits px.choropleth(...) and writes output.json.
• locations needs @SampleColumn("iso_country"), because a generic text column may render nothing.
• color can be filled by type.
• It is verified through Plotly JSON comparison.

Plan: first finish and verify the existing operators one by one with this plan. Then, for future operators: shared fixture by default; when it can't represent the operator's input well, the author provides a small dataset for a more accurate test.