Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE] Rework planner C2R / R2C code with new transition facilities #5764

Open
zhztheplayer opened this issue May 16, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@zhztheplayer
Copy link
Member

zhztheplayer commented May 16, 2024

Transition will stand for interchanges between different in-memory data formats.

The enhancement includes:

  1. Union current C2R / R2C / C2R2C / (future) C2C code together with centralized management for transitions.
    • Remove old rules for inserting C2R / R2C / C2R2C transitions.
    • Remove C2R / R2C creation APIs from SparkPlanExecApi.
  2. Extensible convention API:
    • Convention will be bound to each individual plan operator to describe its supported output in-memory format.
    • Convention will consist of RowType and BatchType
    • BatchType is extensible with predefined API
    • When a new BatchType is defined, the relevant to-row / from-row / to-other-batch / from-other-batch transitions can be defined at the same time.
  3. Insert transitions:
    • Remove old rules for inserting C2R / R2C / C2R2C transitions.
    • Remove code that will remain unused from PlanUtil.
    • Add a new rule to add transitions using the new facility. In the new rule, Gluten will find the cheapest defined transition for two adjacent plan operators.
@zhztheplayer zhztheplayer added the enhancement New feature or request label May 16, 2024
@zhztheplayer zhztheplayer changed the title [CORE] Rework C2R / R2C code with new transition facilities [CORE] Rework planner C2R / R2C code with new transition facilities May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant