fix(ts): support streaming queries in amp query#1854
Conversation
leoyvens
changed the title
amp query
|
@leoyvens wdyt if we instead of extending the Amp TS CLI, we add |
|
@LNSD would be great, but a larger scope than this PR |
fubhy
left a comment
fubhy
left a comment
There was a problem hiding this comment.
@leoyvens @LNSD we are working on a leaner typescript sdk here: https://github.com/edgeandnode/amp-typescript/blob/main/packages/amp
That one also has the correct streamign semantics because it doesn't use the paritally broken (but official) apache arrow (arrow-js) package and instead uses a hand-rolled implementation of arrow purely for decoding to JS (which is what you would normally in JS as we are considering it a leaf end consumer, not an arrow data pipeline processor).
I am aligned with eventually removing the TS/JS implementation of amp from this repository and instead going all in on Rust for CLI, built-in client, etc.
But agreed with Leo that that's a larger scope and for now this is actually reasonable.
|
|
||
| // Force exit to close any open gRPC connections (streaming queries never | ||
| // send EOS so the event loop would otherwise hang indefinitely). | ||
| process.exit(0) |
There was a problem hiding this comment.
If I just remove it, streaming queries with a --limit option hang and the CLI command doesn't terminate.
There was a problem hiding this comment.
If you have suggestions here I'm happy to try them in a follow up
The query command collected all record batches into a single Arrow Table before formatting, which meant streaming (infinite) queries would hang forever waiting for a stream that never ends. This switches to decoding rows as each batch arrives, letting jsonl and pretty formats emit output immediately while table/json still collect but respect --limit to bound the buffering. A process.exit(0) at the end ensures gRPC connections don't keep the event loop alive after a streaming query is interrupted. Signed-off-by: Leonardo Yvens <leoyvens@gmail.com>
leoyvens
force-pushed
the
fix-query-streaming-output
branch
from
6a48ea8 to
8e9bd30
Compare
3 participants
The query command collected all record batches into a single Arrow Table before formatting, which meant streaming queries would hang forever waiting for a stream that never ends. This switches to decoding rows as each batch arrives, letting
jsonlandprettyformats emit output immediately while table/json still collect but respect--limitto bound the buffering.A process.exit(0) at the end ensures gRPC connections don't keep the event loop alive after a streaming query is interrupted.
Tested manually.
@fubhy this is AI code so I'd appreciate a review of Effect usage.