Skip to content

feat(javascript): add support for streaming ingest#4125

Merged
lidavidm merged 1 commit intoapache:mainfrom
kentkwu:gh-4117
Mar 20, 2026
Merged

feat(javascript): add support for streaming ingest#4125
lidavidm merged 1 commit intoapache:mainfrom
kentkwu:gh-4117

Conversation

@kentkwu
Copy link
Contributor

@kentkwu kentkwu commented Mar 19, 2026

Adds streaming ingest support to the Node.js ADBC driver manager.

AdbcConnection:

  • ingestStream(tableName, reader, options?) — streams a RecordBatchReader into a table, handling the full ingest lifecycle

AdbcStatement:

  • bindStream(reader) — streams a RecordBatchReader as bound parameters and leaves execution to the caller

Implementation: ingestStream uses a Rust mpsc channel to bridge the JS push loop and the ADBC execute_update call running on the thread pool. JS serializes each batch to Arrow IPC and sends it to the ChannelBatchReader on the Rust side which deserializes it. The channel is unbounded to avoid blocking the JS event loop.

Test Plan

npm run test

Closes #4117

@amoeba amoeba changed the title feat: add support for streaming ingest feat(javascript): add support for streaming ingest Mar 19, 2026
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is a brand-new IPC reader for every message going to handle things like dictionaries properly? (I guess it would merely be very redundant, we'd send the dictionaries on every message?)

In theory an Arrow implementation could expose the underlying IPC message so you can turn a stream of data into a stream of messages (instead of a stream of serialized files/streams), but I'm not sure if arrow-js + arrow-rs can do this

That said as long as it works we can always improve later...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I explored that and arrow-js currently just doesn't expose a clean API to do it. It would add a chunk of complexity for minimal benefit currently.

At this stage primarily prioritizing functional completeness and then we will continue to improve with benchmarking and performance optimization

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM. Do you think it's worth filing an issue in arrow-js for down the line?

@lidavidm lidavidm merged commit d61179d into apache:main Mar 20, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

javascript: add support for streaming ingestion

2 participants