A result as Arrow, in the bytes every reader takes

[tamnd]

conn.arrow(statement, params, options) runs a statement and gives back the bytes of an Arrow IPC stream, beside the row count, the GQLSTATUS and the notices. tableFromIPC(read.ipc) is the whole of the reading code, and the same call is on a prepared statement.

import { tableFromIPC } from "apache-arrow";

const read = await conn.arrow(`MATCH (p:person) RETURN p.name AS name, p.age AS age`);
const table = tableFromIPC(read.ipc);

Where the translation lives

Not here. zu-arrow in the engine tree is the one answer about what a zu column becomes in Arrow, and the Python client exports through the same code, so a year-month duration is a month interval in both and a node names its table in both. Two copies of it would be two sets of rules about what a duration is, and the second one would drift. What this PR adds is the runtime's half: read the option, run the statement on the threadpool, hand the buffer to V8 without copying it again.

Why bytes and not the C Data Interface

Python takes a pointer, because it can dereference one. Nothing in a JavaScript runtime can, so the value that reaches JavaScript has to be something V8 holds, and the only thing V8 holds that Arrow also speaks is a buffer of IPC bytes. That turns out to be worth having on its own: a worker takes bytes as a transferable rather than cloning them, a fetch body is bytes already, and DuckDB-Wasm reads them as they stand.

What it costs

Over a million rows, from npm run bench:arrow:

one integer column, arrow           8.5 ms       8 ns/row
one integer column, columnar        3.8 ms       4 ns/row
one integer column, rows          276.0 ms     276 ns/row
three columns, arrow               24.1 ms      24 ns/row
three columns, columnar            18.2 ms      18 ns/row
three columns, rows               703.8 ms     704 ns/row

The stream costs about a third more than handing the raw buffers straight over, and about a thirtieth of what building row objects costs. Reading it back is 0.3 ms for those three columns, because tableFromIPC reads the headers and points at the bytes rather than walking them. The batch size makes no measurable difference, which the bench also prints, because the arrays are built whole and a batch is a slice of them.

The pin

The engine pin moves to 0698a4e, the commit zu-arrow landed in. That also makes the columnar block in the README honest again: the sink fills those buffers as the statement runs now, so one integer column read columnar is 2.6 ms against the 38.4 ms the page still claimed, and the gap to the row path is twenty to ninety times rather than six to eight. The numbers were left stale in the last PR because the machine was under load 26 and they could not be measured; it is quiet enough now.

Tests

24 over the new call: every column type including nodes, edges, paths, lists and records, the null row that stays a null, the empty result that still carries its schema, batchRows cutting the stream, the signal, the closed connection, the prepared statement, and the two types Arrow has nowhere to put, which are refused by name rather than translated into something near enough. The whole suite is 363 tests, 354 passing and 9 skipped, check:types is clean and the api report is refreshed.