Skip to content

[Compiler Refactor 4] Introduce new APIs in OperatorDescriptor to support new OpExecConfig interface#1796

Merged
zuozhiw merged 2 commits into
masterfrom
zuozhi-opexec-api
Jan 19, 2023
Merged

[Compiler Refactor 4] Introduce new APIs in OperatorDescriptor to support new OpExecConfig interface#1796
zuozhiw merged 2 commits into
masterfrom
zuozhi-opexec-api

Conversation

@zuozhiw
Copy link
Copy Markdown
Contributor

@zuozhiw zuozhiw commented Jan 18, 2023

This PR is a follow-up of #1791 , where a new OpExecConfig interface is introduced for the new workflow compiler layer. This PR changes the OperatorDescriptor interface to use the new OpExecConfig. Currently, the old and new interfaces exist in parallel. After all migrations are completed, the old interface will be removed.

New interfaces:

  def newOperatorExecutor(operatorSchemaInfo: OperatorSchemaInfo): NewOpExecConfig = {
    throw new UnsupportedOperationException(
      "operator " + operatorIdentifier + " is not migrated to new OpExec API"
    )
  }

  // a logical operator corresponds multiple physical operators (a small DAG)
  def operatorExecutorMultiLayer(operatorSchemaInfo: OperatorSchemaInfo): PhysicalPlan = {
    new PhysicalPlan(List(newOperatorExecutor(operatorSchemaInfo)), List())

Logical operators that correpond to a single physical operator will implement newOperatorExecutor interface.
Logical operators that correpond to a multiple physical operator (such as aggregate) will implement operatorExecutorMultiLayer interface.

@zuozhiw zuozhiw added engine refactor Refactor the code java labels Jan 18, 2023
@zuozhiw zuozhiw requested a review from shengquan-ni January 18, 2023 23:25
@zuozhiw zuozhiw changed the title Introduce new APIs in OperatorDescriptor to support new OpExecConfig interface [Compiler Refactor #4] Introduce new APIs in OperatorDescriptor to support new OpExecConfig interface Jan 18, 2023
@zuozhiw zuozhiw merged commit 6257db3 into master Jan 19, 2023
@zuozhiw zuozhiw deleted the zuozhi-opexec-api branch January 19, 2023 02:53
@Yicong-Huang
Copy link
Copy Markdown
Contributor

@zuozhiw can we add an annotation to indicate the deprecation of the old API? Users may be confused about which API to use now.

@zuozhiw zuozhiw changed the title [Compiler Refactor #4] Introduce new APIs in OperatorDescriptor to support new OpExecConfig interface [Compiler Refactor 4] Introduce new APIs in OperatorDescriptor to support new OpExecConfig interface Jan 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine refactor Refactor the code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants