Skip to content

Add a collectToIpc for zero-copy transmission that isn't version-locked to Arrow #11

@westonpace

Description

@westonpace

Currently when users grab results they are given two less than ideal options. They can use collectToArrow which will return the results as Arrow objects. This requires a very exact match between the library Arrow version and the caller Arrow version. This is even more difficult than the usual headache (bumping user library requires bumping flight library) because JS can easily duplicate dependencies or code paths in the presence of bundlers. For this reason it is often safest to use collectToObjects. However, collectToObjects is not a zero-copy operation and requires the data to be marshaled into JS.

An alternative, which should give the best of both worlds, is to collect into an IPC stream. The caller can then decode the IPC stream with their Arrow library. This will copy all the metadata but none of the data. This should allow callers to use whatever Arrow version they desire.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions