Skip to content

feat(py2udf): analyze control scopes - #8324

Closed
carloea2 wants to merge 3 commits into
apache:mainfrom
carloea2:codex/feat/python-workflow-control-scopes
Closed

feat(py2udf): analyze control scopes#8324
carloea2 wants to merge 3 commits into
apache:mainfrom
carloea2:codex/feat/python-workflow-control-scopes

Conversation

@carloea2

@carloea2 carloea2 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Add lexical-scope indexing and structured control-flow traversal for complete Python statements. The analysis derives deterministic control routes for branching, loops, exceptions, context managers, and nested lexical scopes while keeping rendering concerns outside the analysis layer.

This draft is stacked on #8323. Review this PR's control/scope commit after the preceding drafts; its branch will be rebased as the stack lands.

Any related issues, documentation, discussions?

Related to #8319, #8322, #8323, and discussion #8160.

How was this PR tested?

python -m pytest py2udf/src/test/python/python_to_workflow/mosaic/test_controlwalk.py -q
python -m ruff check py2udf/src/main/python/python_to_workflow/mosaic/analysis py2udf/src/test/python/python_to_workflow/mosaic/test_controlwalk.py
python -m ruff format --check py2udf/src/main/python/python_to_workflow/mosaic/analysis py2udf/src/test/python/python_to_workflow/mosaic/test_controlwalk.py

Result: 2 tests passed; Ruff checks passed.

The parameterized cases cover nested branching, loops, exceptions, context managers, abrupt completion, and lexical-scope boundaries.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

@github-actions github-actions Bot added engine dependencies Pull requests that update a dependency file ddl-change Changes to the TexeraDB DDL pyamber frontend Changes related to the frontend GUI ci changes related to CI docs Changes related to documentations common platform Non-amber Scala service paths amber-integration labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @zyratlo, @aglinxinyuan
    You can notify them by mentioning @zyratlo, @aglinxinyuan in a comment.

@carloea2
carloea2 marked this pull request as ready for review September 1, 2026 23:28
@carloea2
carloea2 force-pushed the codex/feat/python-workflow-control-scopes branch from fb07828 to 880bfc3 Compare September 1, 2026 23:30
@github-actions github-actions Bot removed engine dependencies Pull requests that update a dependency file ddl-change Changes to the TexeraDB DDL pyamber frontend Changes related to the frontend GUI docs Changes related to documentations common platform Non-amber Scala service paths amber-integration labels Sep 1, 2026
@carloea2 carloea2 closed this Sep 1, 2026
@carloea2
carloea2 deleted the codex/feat/python-workflow-control-scopes branch September 1, 2026 23:31
@codecov-commenter

codecov-commenter commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.07%. Comparing base (fc7cfcd) to head (880bfc3).

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8324   +/-   ##
=========================================
  Coverage     94.07%   94.07%           
  Complexity     4814     4814           
=========================================
  Files          1196     1196           
  Lines         48713    48713           
  Branches       5866     5866           
=========================================
  Hits          45825    45825           
  Misses         1436     1436           
  Partials       1452     1452           
Flag Coverage Δ *Carryforward flag
access-control-service 81.00% <ø> (ø)
agent-service 99.32% <ø> (ø)
amber 90.12% <ø> (ø) Carriedforward from fc7cfcd
computing-unit-managing-service 73.67% <ø> (ø)
config-service 87.00% <ø> (ø)
file-service 87.91% <ø> (ø)
frontend 96.80% <ø> (ø)
notebook-migration-service 79.31% <ø> (ø)
pyamber 98.18% <ø> (ø)
workflow-compiling-service 77.19% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 9 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main fc7cfcd benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 370 0.226 25,950/39,564/39,564 us 🔴 +16.4% / 🔴 +148.8%
🔴 bs=100 sw=10 sl=64 796 0.486 127,423/144,069/144,069 us 🔴 +8.0% / 🔴 +30.2%
🔴 bs=1000 sw=10 sl=64 909 0.555 1,096,993/1,182,492/1,182,492 us 🔴 +5.8% / 🔴 +12.2%
Baseline details

Latest main fc7cfcd from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 370 tuples/sec 383 tuples/sec 750.65 tuples/sec -3.4% -50.7%
bs=10 sw=10 sl=64 MB/s 0.226 MB/s 0.234 MB/s 0.458 MB/s -3.4% -50.7%
bs=10 sw=10 sl=64 p50 25,950 us 27,212 us 13,049 us -4.6% +98.9%
bs=10 sw=10 sl=64 p95 39,564 us 34,004 us 15,902 us +16.4% +148.8%
bs=10 sw=10 sl=64 p99 39,564 us 34,004 us 19,184 us +16.4% +106.2%
bs=100 sw=10 sl=64 throughput 796 tuples/sec 844 tuples/sec 960.3 tuples/sec -5.7% -17.1%
bs=100 sw=10 sl=64 MB/s 0.486 MB/s 0.515 MB/s 0.586 MB/s -5.6% -17.1%
bs=100 sw=10 sl=64 p50 127,423 us 117,977 us 104,199 us +8.0% +22.3%
bs=100 sw=10 sl=64 p95 144,069 us 136,693 us 110,645 us +5.4% +30.2%
bs=100 sw=10 sl=64 p99 144,069 us 136,693 us 120,888 us +5.4% +19.2%
bs=1000 sw=10 sl=64 throughput 909 tuples/sec 927 tuples/sec 993.27 tuples/sec -1.9% -8.5%
bs=1000 sw=10 sl=64 MB/s 0.555 MB/s 0.566 MB/s 0.606 MB/s -1.9% -8.5%
bs=1000 sw=10 sl=64 p50 1,096,993 us 1,078,784 us 1,012,191 us +1.7% +8.4%
bs=1000 sw=10 sl=64 p95 1,182,492 us 1,117,641 us 1,054,057 us +5.8% +12.2%
bs=1000 sw=10 sl=64 p99 1,182,492 us 1,117,641 us 1,081,313 us +5.8% +9.4%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,541.01,200,128000,370,0.226,25949.58,39564.20,39564.20
1,100,10,64,20,2512.41,2000,1280000,796,0.486,127423.05,144069.11,144069.11
2,1000,10,64,20,22006.42,20000,12800000,909,0.555,1096993.48,1182491.60,1182491.60

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci changes related to CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants