Skip to content

Conversation

@Ted-Jiang
Copy link
Member

@Ted-Jiang Ted-Jiang commented Mar 14, 2022

Which issue does this PR close?

Closes #2007 .

Rationale for this change

Now support more ScalarFunction in ballista. like "random"
start 1 scheduler + 1 executor
run client with cargo run --features=ballista --bin datafusion-cli -- --host localhost --port 50050
Before:

select * from lineitem where l_quantity = random()*10;

Internal("failed to serialize logical plan: General(\"Unsupported scalar function Random\")")

Now:

select * from lineitem where l_quantity = trunc(random() * 9 + 1) limit 3;
+------------+-----------+-----------+--------------+------------+-----------------+------------+-------+--------------+--------------+------------+--------------+---------------+-------------------+------------+-------------------------+
| l_orderkey | l_partkey | l_suppkey | l_linenumber | l_quantity | l_extendedprice | l_discount | l_tax | l_returnflag | l_linestatus | l_shipdate | l_commitdate | l_receiptdate | l_shipinstruct    | l_shipmode | l_comment               |
+------------+-----------+-----------+--------------+------------+-----------------+------------+-------+--------------+--------------+------------+--------------+---------------+-------------------+------------+-------------------------+
| 7          | 145243    | 7758      | 2            | 9          | 11594.16        | 0.08       | 0.08  | N            | O            | 1996-02-01 | 1996-03-02   | 1996-02-19    | TAKE BACK RETURN  | SHIP       | es. instructions        |
| 35         | 120896    | 8433      | 3            | 7          | 13418.23        | 0.06       | 0.04  | N            | O            | 1996-01-19 | 1995-12-22   | 1996-01-29    | NONE              | MAIL       |  the carefully regular  |
| 163        | 192642    | 5162      | 4            | 5          | 8673.2          | 0.02       | 0     | N            | O            | 1997-11-17 | 1997-10-09   | 1997-12-05    | DELIVER IN PERSON | TRUCK      |  must belie             |
+------------+-----------+-----------+--------------+------------+-----------------+------------+-------+--------------+--------------+------------+--------------+---------------+-------------------+------------+-------------------------+
3 rows in set. Query took 1.897 seconds.

What changes are included in this PR?

Are there any user-facing changes?

@liukun4515
Copy link
Contributor

Thanks, @Ted-Jiang I will go through this pr later.

ScalarFunction::Sin => Ok(sin((&args[0]).try_into()?)),
ScalarFunction::Cos => Ok(cos((&args[0]).try_into()?)),
ScalarFunction::Tan => Ok(tan((&args[0]).try_into()?)),
// ScalarFunction::Asin => Ok(asin(&args[0]).try_into()?)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the commented-out Asin, Acos, Concat, Array, Nullif, and md5 are removed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opps, forgot the deleted line, have added.

Copy link
Contributor

@liukun4515 liukun4515 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
It's would be better if we had the tests for the build-in function on the ballista side.
@Ted-Jiang Can you help to add some tests for this in the follow-up pull request?

@doki23
Copy link
Contributor

doki23 commented Mar 16, 2022

may need a follow-up pr when #1969 merged

Copy link
Member

@yjshen yjshen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Ted-Jiang, LGTM.

@yjshen yjshen merged commit dbc5815 into apache:master Mar 16, 2022
@yjshen yjshen added the api change Changes the API exposed to users of the crate label Mar 16, 2022
@yjshen yjshen mentioned this pull request Mar 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Ballista] Support more ScalarFunction in Ballista

4 participants