Skip to content

fix: fold variable expressions before remote run - #24800

Merged
XuPeng-SH merged 2 commits into
matrixorigin:mainfrom
LeftHandCold:fix/var-expr-remote-run-main
Jun 4, 2026
Merged

fix: fold variable expressions before remote run#24800
XuPeng-SH merged 2 commits into
matrixorigin:mainfrom
LeftHandCold:fix/var-expr-remote-run-main

Conversation

@LeftHandCold

@LeftHandCold LeftHandCold commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Fixes #24799.

Summary

  • fold @@ variable expressions on the source CN before a scope is shipped to a remote CN, preserving the remote execution path
  • deep-copy expression containers before folding and write copied expressions back to the parent field/slice/map, avoiding shared-plan mutation, data races, and prepare-plan cache pollution
  • cover exported expression fields, slices/maps, agg argument expressions, and lockRows hidden expressions
  • keep the VarExpressionExecutor nil-resolver guard so any missed case returns a normal error instead of panicking

Testing

  • GOCACHE=/private/tmp/mo-gocache-var-expr-main-copy GOFLAGS=-mod=mod go test ./pkg/sql/colexec ./pkg/sql/compile -run 'TestVarExpressionExecutorWithoutResolveVariableFunc|TestScopeContainsVarExpr|TestFoldVarExprs' -count=1 is blocked in this local main checkout by existing usearch cgo build errors: could not determine what C.usearch_* refers to.

Copilot AI review requested due to automatic review settings June 3, 2026 06:49
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a remote-run failure mode where variable expressions (plan.Expr_V) can panic (or now error) when executed on a reconstructed remote process.Process that does not carry a ResolveVariableFunc. It adds scope-level detection to force local execution for pipelines containing variable expressions, and makes variable-expression evaluation defensive by returning an error instead of panicking.

Changes:

  • Convert remote-run scopes to local run when the scope contains variable expressions.
  • Make VarExpressionExecutor.Eval return a normal internal error when ResolveVariableFunc is not set.
  • Add unit tests for scope variable-expression detection and nil resolver behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
pkg/sql/compile/scope.go Falls back to local MergeRun when a remote-run scope is detected to contain variable expressions.
pkg/sql/compile/remote_expr.go Adds reflection-based detection for plan.Expr_V within Scope/Source/operator trees.
pkg/sql/compile/remote_expr_test.go Adds unit tests for scope variable-expression detection.
pkg/sql/colexec/evalExpression.go Adds nil check around proc.GetResolveVariableFunc() to avoid panic.
pkg/sql/colexec/evalExpression_test.go Adds coverage for nil ResolveVariableFunc behavior.

Comment thread pkg/sql/compile/remote_expr.go
Comment thread pkg/sql/compile/remote_expr.go
@LeftHandCold
LeftHandCold force-pushed the fix/var-expr-remote-run-main branch from 978f06a to d0118b1 Compare June 3, 2026 07:31
@LeftHandCold
LeftHandCold force-pushed the fix/var-expr-remote-run-main branch from d0118b1 to 374c84d Compare June 3, 2026 08:29
@LeftHandCold

Copy link
Copy Markdown
Contributor Author

已按后续 review 调整:RemoteRun 前不再切 MergeRun,而是在 source CN 对 @@ var expr 做 fold;fold 前 DeepCopyExpr 并写回父 field/slice/map,避免并发 remote scope 共享表达式导致 data race,也避免 prepare plan 被第一次执行污染。getter-only 折叠路径已移除,agg args/lockRows 改成 rewriter 写回;对应 copy-on-write 回归测试已补。

@aunjgr aunjgr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Practical fix for remote CN variable resolution. Same change across three branches (#24800 main, #24801 4.0-dev, #24802 3.0-dev).

Approach: Before sending a scope to a remote CN, walk the operator tree via reflection to find VarRef expressions, evaluate them locally (where the session variable context exists), and replace with constants. Uses scopeContainsVarExpr as a fast-path guard so the reflection-based folding only runs when needed.

remote_expr.go (490 new lines): reflection-based traversal of operator structs looking for *plan.Expr fields ([]*plan.Expr, []plan.Expr, and operator-base fields are handled). Two custom interfaces for operators that need special treatment:

  • argExpressionsGetter/Rewriter for AggFuncExecExpression (aggregate arg expressions)
  • lockRowsExpressionsGetter/Rewriter for LockOp (lock row filter expressions)

Other changes:

  • evalExpression.go: nil guard on resolveVariableFunc — returns a clear error instead of nil-deref on remote CN
  • aggexec/types.go: implements RewriteArgExpressions for AggFuncExecExpression
  • lockop/lock_op.go: implements GetLockRowsExpressions / RewriteLockRowsExpressions for LockOp
  • scope.go: removes dead VarExpr field from Scope (unused, confirmed by grep of s.VarExpr)

Reflection safety: only plan.Expr types are matched (via planExprPtrType); the walk handles pointers, slices, and operator-base embedding. replaceVarRefExpr only fires on actual VarRef nodes, so nothing gets accidentally constant-folded.

LGTM. Please also merge #24801 and #24802 (same fix, different target branches).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/L Denotes a PR that changes [500,999] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote pipeline with variable expression panics when ResolveVariableFunc is missing

6 participants