Switch to Arrow2/Parquet2

[kylebarron]

Seems to be a bit faster (and safer Rust), but most notably, the generated IPC just works out of the box in Arrow JS.

Ignore the ugly newbie rust code 😂

[kylebarron]

the generated IPC just works out of the box in Arrow JS.

Well actually it does in most of the cases tested, but not "./data/2-partition-brotli.parquet";. There it gives

message.js:102 Uncaught (in promise) Error: Expected to read 1901288 metadata bytes, but only read 644.

When trying to read in JS 😢 .

[kylebarron]

In search of a large test file, I found s3://ookla-open-data/parquet/performance/type=fixed/year=2021/quarter=1/2021-01-01_performance_fixed_tiles.parquet (from https://registry.opendata.aws/speedtest-global-performance/). The Parquet file is 549MB, which expands to 1.52GB in memory (judging from Pyarrow). The whole file crashed WASM with panicked at 'capacity overflow'.

The first 10 row groups (out of 18) are a 316MB Parquet file. That's 888MiB according to Pyarrow. It takes about 8.5s to return the decoded bytes to JS. It takes about 2s to load the bytes using Pyarrow. Not shabby at all, and it works! 🎉

Selecting the first n row groups:

import pyarrow.parquet as pq
f = pq.ParquetFile('2021-01-01_performance_fixed_tiles.parquet')
with pq.ParquetWriter('part.parquet', schema=f.schema_arrow) as writer:
    for i in range(10):
        writer.write_table(f.read_row_group(i))

[kylebarron]

Well actually it does in most of the cases tested, but not "./data/2-partition-brotli.parquet";. There it gives

message.js:102 Uncaught (in promise) Error: Expected to read 1901288 metadata bytes, but only read 644.

When trying to read in JS 😢 .

This appears to be a JS error again. This file loads fine in Python with pa.ipc.open_file() but gives the same error as in JS with pa.ipc.open_stream(). So it's likely JS's arrow.tableFromIPC is incorrectly deducing it as an IPC file and not an IPC stream.