Fix bug in size estimation of array buffers

[emilk]

An off-by-one bug in estimated_bytes_size caused the size estimations of Tensors to completely ignore the actual payload, causing the memory consumption of images to be vastly underestimated.

I decided to rewrite the code to be more clear for future readers.

This discovered a difference in how validity bitmaps are written by Rust and Python+C++, fixed in a828441.

To help find this problem a lot of work was also put into improving the output of scripts/ci/run_e2e_roundtrip_tests.py.

• Discovered while working on Show per-entity data rate and volume #2435

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)
• I have tested demo.rerun.io (if applicable)

• PR Build Summary
• Docs preview
• Examples preview

[Wumpf]

👍

[jleibs]

Nice catch

[emilk]

This seems to be catching a bug in the e2e tests now! That's good, I guess :) But needs debugging…

I should probably a dd a unit-test also to make sure the new code is correct, and won't regress.

[emilk]

It is translation_and_mat3x3/identity that fails:

and with a slightly different diff each time:

[emilk]

So I've found the difference: the python version writes a validity bitmaps where the rust version doesn't. Maybe the rust arrow library omits them when they are all set or something. I will dig deeper. Still, this is a real difference in the data that rerun compare didn't catch until now. Perhaps though it is a difference that should be ignored.

[emilk]

So on the Rust size, TranslationAndMat3x3 has this datatype (crates/re_types/src/datatypes/translation_and_mat3x3.rs):

DataType::Struct(vec![
    Field {
        name: "translation".to_owned(),
        data_type: <crate::datatypes::Vec3D>::to_arrow_datatype(),
        is_nullable: true,
        metadata: [].into(),
    },
    Field {
        name: "matrix".to_owned(),
        data_type: <crate::datatypes::Mat3x3>::to_arrow_datatype(),
        is_nullable: true,
        metadata: [].into(),
    },
    Field {
        name: "from_parent".to_owned(),
        data_type: DataType::Boolean,
        is_nullable: false,
        metadata: [].into(),
    },
])

while Python has this (rerun_py/rerun_sdk/rerun/_rerun2/datatypes/transform3d.py):

pa.struct(
    [
        pa.field(
            "translation",
            pa.list_(pa.field("item", pa.float32(), nullable=False, metadata={}), 3),
            nullable=True,
            metadata={},
        ),
        pa.field(
            "matrix",
            pa.list_(pa.field("item", pa.float32(), nullable=False, metadata={}), 9),
            nullable=True,
            metadata={},
        ),
        pa.field("from_parent", pa.bool_(), nullable=False, metadata={}),
    ]
),

EDIT: @jleibs pointed out this is not wrong

[jleibs]

Notice how Rust sets the first fields as nullable, but Python doesn't!

I think you're misreading the python. The field that's set non-nullible there is the inner nested type, not the outer type.

pa.field("matrix", pa.list_(pa.field("item", pa.float32(), False, {}), 9), True, {})

For clarity it would be nice if this were re-implemented recursively as:

pa.field("matrix", Vec3D.data_type(), True, {})

But unless the inner types don't match those look like they are all doing the same thing.

[emilk]

Oh, you're right… I got confused by how Rust and C++ calls out to other functions for the inner datatype, while Python repeats/inlines it. Nevermind then.

[emilk]

The Python encoder is outputting validities with all zeroes, but the Rust and C++ arrow encoder omits the validity.

Maybe it is an optimization in Rust and C++ to only output the validity if it is non-zero? That works if the encoder interprets a missing validity as "all nulls".

From https://arrow.apache.org/docs/format/Columnar.html#sparse-union:

Arrays having a 0 null count may choose to not allocate the validity bitmap; how this is represented depends on the implementation (for example, a C++ implementation may represent such an “absent” validity bitmap using a NULL pointer). Implementations may choose to always allocate a validity bitmap anyway as a matter of convenience. Consumers of Arrow arrays should be ready to handle those two possibilities.

This sounds to me like an omitted validity map means "all set = no nulls"… I'm giving up for today.

[jleibs]

The Python encoder is outputting validities with all zeroes, but the Rust and C++ arrow encoder omits the validity.

This doesn't match what I'm seeing. For me, Python & C++ both output the same encoding and Rust is the one that's different.