Arrow Array Stream import/export?

[eitsupi]

I wanted to create a MemTable from an Array Stream, but realized that MemTable does not expose the internal Arrow data and I can only access the original data by executing a query.

datafusion/datafusion-examples/examples/memtable.rs

Lines 30 to 52 in f807947

	async fn main() -> Result<()> {
	let mem_table = create_memtable()?;
	// create local execution context
	let ctx = SessionContext::new();
	// Register the in-memory table containing the data
	ctx.register_table("users", Arc::new(mem_table))?;
	let dataframe = ctx.sql("SELECT * FROM users;").await?;
	timeout(Duration::from_secs(10), async move {
	let result = dataframe.collect().await.unwrap();
	let record_batch = result.first().unwrap();
	assert_eq!(1, record_batch.column(0).len());
	dbg!(record_batch.columns());
	})
	.await
	.unwrap();
	Ok(())
	}

Which Struct that DataFusion has would be ideal if I want to input/output data?
(I mean I was able to input data into MemTable and was hoping to be able to output the same, but apparently not, so I'm stumped)

[jonahgao]

Do you want to access the data inside the Memtable directly without using a query?
I'm not entirely sure, but I didn't find such a structure; it seems you might need to consider customizing a TableProvider based on Memtable.

[eitsupi]

Thanks for your reply.
Finally, I judged it reasonable to define a simple Struct with C Stream interface.

struct Table {
    pub schema: std::sync::Arc<arrow::datatypes::Schema>,
    pub batches: Vec<datafusion::arrow::array::RecordBatch>,
}