Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-10766: [Rust] [Parquet] Compute nested list definitions #9240

Closed
wants to merge 2 commits into from
Closed

ARROW-10766: [Rust] [Parquet] Compute nested list definitions #9240

wants to merge 2 commits into from

Commits on Jan 20, 2021

  1. ARROW-10766: [Rust] [Parquet] Compute nested list definitions 

    This mainly computes definition and repetition leves for lists.
It also partially adds deeply nested write support.
I am however going to complete this in a separate PR.

This has really been challenging because we can't roundtrip without nested writers,
so it's taken me months to complete.
In the process, I've had to rely on using Spark to verify my work.

This PR is also not optimised. I've left TODOs in a few places (sparingly).
The biggest next step is to remove array_mask: Vec<u8> and replace it with a bitpacked vector to save memory.
    @nevi-me
    nevi-me committed Jan 20, 2021
    Configuration menu
    Copy the full SHA
    df553ca View commit details
    Browse the repository at this point in the history

  2. clean up

    @nevi-me
    nevi-me committed Jan 20, 2021
    Configuration menu
    Copy the full SHA
    a59613b View commit details
    Browse the repository at this point in the history