Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(flow): support datafusion scalar function #4142

Merged
merged 11 commits into from
Jun 18, 2024

Conversation

discord9
Copy link
Contributor

@discord9 discord9 commented Jun 14, 2024

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

support datafusion scalar function

Please explain IN DETAIL what the changes are in this PR and why they are needed:

support datafusion scalar function by let datafusion decode substrait proto, and hence build df PhysicalExpr to evaluate function

  • added DfScalarFunction which hold both impl of that scalar function and necessary data to ser/de it
  • added path in transform module to translate scalar function to df scalar function when no other functions are available

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.

@github-actions github-actions bot added the docs-not-required This change does not impact docs. label Jun 14, 2024
@discord9 discord9 marked this pull request as ready for review June 14, 2024 09:16
@discord9 discord9 requested review from zhongzc, waynexia and a team as code owners June 14, 2024 09:16
Copy link

codecov bot commented Jun 14, 2024

Codecov Report

Attention: Patch coverage is 71.21771% with 78 lines in your changes missing coverage. Please review.

Project coverage is 84.60%. Comparing base (9473daa) to head (a1ede1a).
Report is 18 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4142      +/-   ##
==========================================
- Coverage   85.46%   84.60%   -0.87%     
==========================================
  Files         994     1023      +29     
  Lines      174582   179538    +4956     
==========================================
+ Hits       149204   151894    +2690     
- Misses      25378    27644    +2266     

Copy link
Contributor

@zhongzc zhongzc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unit tests required

src/flow/src/expr/scalar.rs Outdated Show resolved Hide resolved
src/flow/src/expr/scalar.rs Outdated Show resolved Hide resolved
src/flow/src/expr/scalar.rs Show resolved Hide resolved
src/flow/src/transform/expr.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, we need some tests for it.

src/flow/Cargo.toml Outdated Show resolved Hide resolved
src/flow/src/expr/scalar.rs Show resolved Hide resolved
src/flow/src/expr/scalar.rs Outdated Show resolved Hide resolved
src/flow/src/expr/scalar.rs Outdated Show resolved Hide resolved
src/flow/src/expr/scalar.rs Outdated Show resolved Hide resolved
src/flow/src/expr/scalar.rs Show resolved Hide resolved
src/flow/src/expr/scalar.rs Outdated Show resolved Hide resolved
src/flow/src/transform/expr.rs Outdated Show resolved Hide resolved
src/flow/src/transform/expr.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@discord9 discord9 enabled auto-merge June 18, 2024 12:34
@discord9 discord9 added this pull request to the merge queue Jun 18, 2024
Merged via the queue into GreptimeTeam:main with commit cd9705c Jun 18, 2024
49 checks passed
@discord9 discord9 deleted the flow_more_scalar_func branch June 18, 2024 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required This change does not impact docs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants