Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Minor: Improve documentation for PartitionStream and StreamingTableExec #8035

Merged
merged 4 commits into from
Nov 3, 2023
Merged

Minor: Improve documentation for PartitionStream and StreamingTableExec #8035

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e91f0b8
Minor: Improve documentation for `PartitionStream` and `StreamingTab…
alamb Nov 2, 2023
76db218
fmt
alamb Nov 2, 2023
6b955b4
fmt
alamb Nov 2, 2023
4117c1f
Merge remote-tracking branch 'apache/main' into alamb/streaming_docs
alamb Nov 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions datafusion/physical-plan/src/streaming.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.

//! Execution plan for streaming [`PartitionStream`]
//! Generic plans for deferred execution: [`StreamingTableExec`] and [`PartitionStream`]

use std::any::Any;
use std::sync::Arc;
Expand All @@ -35,6 +35,10 @@ use futures::stream::StreamExt;
use log::debug;

/// A partition that can be converted into a [`SendableRecordBatchStream`]
///
/// Combined with [`StreamingTable`], you can use this trait to implement
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is the core usecase -- can you confirm @tustvold ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, it avoids a lot of boilerplate for the simple use-case

/// [`ExecutionPlan`] for a custom source with less boiler plate than
/// implementing `ExecutionPlan` directly for many use cases.
pub trait PartitionStream: Send + Sync {
/// Returns the schema of this partition
fn schema(&self) -> &SchemaRef;
Expand All @@ -43,7 +47,10 @@ pub trait PartitionStream: Send + Sync {
fn execute(&self, ctx: Arc<TaskContext>) -> SendableRecordBatchStream;
}

/// An [`ExecutionPlan`] for [`PartitionStream`]
/// An [`ExecutionPlan`] for one or more [`PartitionStream`]s.
///
/// If your source can be represented as one or more [`PartitionStream`]s, you can
/// use this struct to implement [`ExecutionPlan`].
pub struct StreamingTableExec {
partitions: Vec<Arc<dyn PartitionStream>>,
projection: Option<Arc<[usize]>>,
Expand Down
Loading