Skip to content

JSON decoder for RunEndEncoded accepts nulls even though the values field is non-nullable #10478

Description

@mzabaluev

Describe the bug

The implementation for REE decoder forces the child field's decoder to accept nulls even though the Arrow schema for the values field may declare it non-nullable. As the REE encoding has no container-level null mask, this may smuggle null values into the values array that must not have them.

Furthermore, even if the REE field itself is non-nullable, the flag passed down to construct the child decoder is ignored by e.g. StringArrayDecoder. The resulting nulls are not invalidated by a post-hoc check the way StructArrayDecoder does.

To Reproduce

Repro test case:

    #[test]
    fn test_read_run_end_encoded_nulls_in_non_nullable_values() {
        // A `RunArray` has no validity buffer of its own, so unlike a struct child
        // there is no parent-level mask that could make these nulls representable.
        // Declaring the enclosing field nullable does not change that.
        let ree_type = DataType::RunEndEncoded(
            Arc::new(Field::new("run_ends", DataType::Int32, false)),
            Arc::new(Field::new("values", DataType::Utf8, false)),
        );
        let schema = Arc::new(Schema::new(vec![Field::new(
            "a",
            ree_type,
            true,
        )]));

        // An explicit null, and a missing field, should be both rejected.
        for buf in [
            r#"
            {"a": "x"}
            {"a": null}
            {"a": "y"}
            "#,
            r#"
            {"a": "x"}
            {}
            {"a": "y"}
            "#,
        ] {
            let mut decoder = ReaderBuilder::new(schema.clone()).build_decoder().unwrap();
            let res = decoder
                .decode(buf.as_bytes())
                .and_then(|_| decoder.flush());
            assert!(res.is_err());
        }
    }

Expected behavior

The schema violation should produce an error.

Additional context

Note that this is a breaking change, since the current implementation may tolerate nullable inputs, even if they produce non-compliant outputs.

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions