Describe the bug
The implementation for REE decoder forces the child field's decoder to accept nulls even though the Arrow schema for the values field may declare it non-nullable. As the REE encoding has no container-level null mask, this may smuggle null values into the values array that must not have them.
Furthermore, even if the REE field itself is non-nullable, the flag passed down to construct the child decoder is ignored by e.g. StringArrayDecoder. The resulting nulls are not invalidated by a post-hoc check the way StructArrayDecoder does.
To Reproduce
Repro test case:
#[test]
fn test_read_run_end_encoded_nulls_in_non_nullable_values() {
// A `RunArray` has no validity buffer of its own, so unlike a struct child
// there is no parent-level mask that could make these nulls representable.
// Declaring the enclosing field nullable does not change that.
let ree_type = DataType::RunEndEncoded(
Arc::new(Field::new("run_ends", DataType::Int32, false)),
Arc::new(Field::new("values", DataType::Utf8, false)),
);
let schema = Arc::new(Schema::new(vec![Field::new(
"a",
ree_type,
true,
)]));
// An explicit null, and a missing field, should be both rejected.
for buf in [
r#"
{"a": "x"}
{"a": null}
{"a": "y"}
"#,
r#"
{"a": "x"}
{}
{"a": "y"}
"#,
] {
let mut decoder = ReaderBuilder::new(schema.clone()).build_decoder().unwrap();
let res = decoder
.decode(buf.as_bytes())
.and_then(|_| decoder.flush());
assert!(res.is_err());
}
}
Expected behavior
The schema violation should produce an error.
Additional context
Note that this is a breaking change, since the current implementation may tolerate nullable inputs, even if they produce non-compliant outputs.
Describe the bug
The implementation for REE decoder forces the child field's decoder to accept nulls even though the Arrow schema for the values field may declare it non-nullable. As the REE encoding has no container-level null mask, this may smuggle null values into the values array that must not have them.
Furthermore, even if the REE field itself is non-nullable, the flag passed down to construct the child decoder is ignored by e.g.
StringArrayDecoder. The resulting nulls are not invalidated by a post-hoc check the wayStructArrayDecoderdoes.To Reproduce
Repro test case:
Expected behavior
The schema violation should produce an error.
Additional context
Note that this is a breaking change, since the current implementation may tolerate nullable inputs, even if they produce non-compliant outputs.