[GLUTEN-1858][CORE] Add PlanOneRowRelation to make gluten work with OneRowRelation#1859
Conversation
|
Run Gluten Clickhouse CI |
ulysses-you
changed the title
|
cc @zhztheplayer @zhouyuan thank you |
ulysses-you
force-pushed
the
one-row-relation
branch
from
a4f31b8 to
aa7e536
Compare
|
Run Gluten Clickhouse CI |
ulysses-you
force-pushed
the
one-row-relation
branch
2 times, most recently
from
0cabf48 to
671610d
Compare
|
Run Gluten Clickhouse CI |
| val plan = df.queryExecution.executedPlan | ||
| assert(plan.find(_.isInstanceOf[RDDScanExec]).isDefined) | ||
| assert(plan.find(_.isInstanceOf[ProjectExecTransformer]).isDefined) | ||
| assert(plan.find(_.isInstanceOf[GlutenRowToArrowColumnarExec]).isDefined) |
There was a problem hiding this comment.
There's a refactor on class name on main branch
| assert(plan.find(_.isInstanceOf[GlutenRowToArrowColumnarExec]).isDefined) | |
| assert(plan.find(_.isInstanceOf[RowToArrowColumnarExec]).isDefined) |
zhouyuan
left a comment
zhouyuan
left a comment
There was a problem hiding this comment.
@ulysses-you can you please do a rebase?
ulysses-you
force-pushed
the
one-row-relation
branch
from
671610d to
caf769a
Compare
|
Run Gluten Clickhouse CI |
|
thank you @zhouyuan , rebased |
ulysses-you
force-pushed
the
one-row-relation
branch
from
caf769a to
e898727
Compare
|
Run Gluten Clickhouse CI |
ulysses-you
force-pushed
the
one-row-relation
branch
from
e898727 to
7fec6e0
Compare
|
Run Gluten Clickhouse CI |
|
@zhouyuan any idea about the failed test ? I'm not sure what's wrong |
|
@ulysses-you yes, the failure seems due to one change on velox/substrait this morning, fix up ASAP |
ulysses-you
force-pushed
the
one-row-relation
branch
from
7fec6e0 to
4a3e2bb
Compare
|
Run Gluten Clickhouse CI |
ulysses-you
force-pushed
the
one-row-relation
branch
from
4a3e2bb to
f5b00b3
Compare
|
Run Gluten Clickhouse CI |
|
@zhouyuan all tests passed ! |
|
@ulysses-you Thank you for improving on this part! Yes, in the UT gluten should disabled ConstantFolding already. CC: @PHILO-HE |
|
yeas, I think it can help improve test coverage a bit. Some tests in Spark are written with |
2 participants
What changes were proposed in this pull request?
This pr adds a new rule
PlanOneRowRelationwhich inject a fake schema forOneRowRelation. As gluten does not support work with empty output relation.BTW, if we want to validate data result with vanilla spark and gluten, please disable constant folding
set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding(Fixes: #1858)
How was this patch tested?
add test