ARROW-9683: [Rust][DataFusion] Add debug printing to physical plans and associated types #7925
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
commented
Aug 10, 2020
| @@ -61,6 +73,12 @@ pub struct DatasourcePartition { | |||
| batch_iter: Arc<Mutex<dyn RecordBatchReader + Send + Sync>>, | |||
| } | |||
|
|
|||
| impl Debug for DatasourcePartition { | |||
There was a problem hiding this comment.
RecordBatchReader does not implement Debug so had to implement directly here.
| @@ -110,6 +112,12 @@ struct ParquetPartition { | |||
| iterator: Arc<Mutex<dyn RecordBatchReader + Send + Sync>>, | |||
| } | |||
|
|
|||
| impl Debug for ParquetPartition { | |||
| fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { | |||
| f.debug_struct("ParquetPartition").finish() | |||
There was a problem hiding this comment.
As above, this is not #derive because RecordBatchReader does not implement Debug
| .field("name", &self.name) | ||
| .field("args", &self.args) | ||
| .field("return_type", &self.return_type) | ||
| .field("fun", &"<FUNC>") |
There was a problem hiding this comment.
The ScanarUdf link is typedef'd to a function pointer and does not implement Debug leading to the two implementations in this file
…nd associated types
alamb
force-pushed
the
alamb/ARROW-9683-debug
branch
from
e21b7a5
to
9a59730
Compare
|
Rebased to pick up 37ee600 |
andygrove
approved these changes
Aug 10, 2020
nevi-me
approved these changes
Aug 11, 2020
GeorgeAp
pushed a commit
to sirensolutions/arrow
that referenced
this pull request
Jun 7, 2021
…nd associated types
For ARROW-9653, I was trying to debug the execution plan and I would have found it easier if there had been a way to display the execution plan to better understand and isolate the issue. This would also be nice to have as part of EXPLAIN plan functionality in ARROW-9654
In general, for debugging purposes, we would like to be able to dump out an execution plan. To do so in the idiomatic rust way, I made `ExecutionPlan` also implement `std::fmt::Debug` and then followed `rustc` guidance until I got everything that was needed
Here is an example plan for `"SELECT c1, c2, MIN(c3) FROM aggregate_test_100 GROUP BY c1, c2"` when printed using `println!("{:#?}", plan)`:
```
physical plan is HashAggregateExec {
group_expr: [
Column {
name: "c1",
},
Column {
name: "c2",
},
],
aggr_expr: [
Min {
expr: Column {
name: "c3",
},
},
],
input: DataSourceExec {
schema: Schema {
fields: [
Field {
name: "c1",
data_type: Utf8,
nullable: false,
dict_id: 0,
dict_is_ordered: false,
},
Field {
name: "c2",
data_type: UInt32,
nullable: false,
dict_id: 0,
dict_is_ordered: false,
},
Field {
name: "c3",
data_type: Int8,
nullable: false,
dict_id: 0,
dict_is_ordered: false,
},
],
metadata: {},
},
partitions.len: 1,
},
schema: Schema {
fields: [
Field {
name: "c1",
data_type: Utf8,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
},
Field {
name: "c2",
data_type: UInt32,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
},
Field {
name: "MIN(c3)",
data_type: Int64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
},
],
metadata: {},
},
}
```
Closes apache#7925 from alamb/alamb/ARROW-9683-debug
Authored-by: alamb <andrew@nerdnetworks.org>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For ARROW-9653, I was trying to debug the execution plan and I would have found it easier if there had been a way to display the execution plan to better understand and isolate the issue. This would also be nice to have as part of EXPLAIN plan functionality in ARROW-9654
In general, for debugging purposes, we would like to be able to dump out an execution plan. To do so in the idiomatic rust way, I made
ExecutionPlanalso implementstd::fmt::Debugand then followedrustcguidance until I got everything that was neededHere is an example plan for
"SELECT c1, c2, MIN(c3) FROM aggregate_test_100 GROUP BY c1, c2"when printed usingprintln!("{:#?}", plan):