Reading Arrow NullVector

[Kopilov]

Apache Arrow files might contain NullVector values (as result of saving null-infilled column in other libraries and languages without static types and target schema). From this PR they will be correctly read by Kotlin DataFrame instead of crushing. Also we can make saving to NullVectors, should we?

Among others, Arrow itself is upgraded to last stable version (14.0.2) and #428 problem is fixed for Arrow writing by replacing original hasNulls function with custom explicit checking.

[Jolanrensen]

Please revert this line as it's less efficient. The hasNulls()/isMarkedNullable is set upon column creation and since columns are immutable we can safely assume all values are not null if hasNulls() == false. Maybe we could only run this check if strictNullable == true, wdyt @koperagen?

[Kopilov]

OK, I have removed that commit on rebasing to new master. But we still can specify nullable type for not-null values and vice versa (look first message in #428, that test is reproduceable just now).

Maybe we could only run this check if strictNullable == true

Sounds good, I can make this in separate PR. Or we should fix #428 totally in any another way (such as keeping "nullable" for types and "hasNulls" for values independently). You were agree that hasNulls() should only return true if there are actually nulls in the column.

[Jolanrensen]

We're indeed still working on figuring out a good way to fix #428 so that the cases you gave are not possible anymore. That way hasNulls() == isMarkedNullable at all times. The difficult part is finding a good balance between performance and safety. We don't want to have to scan an entire list of data if a user can ensure it has or has no nulls...

[Kopilov]

Merge branch 'master' into NullVector

Should we apply git rebase instead (to avoid spaghetti-like history)?

[Kopilov]

Rebased

[Jolanrensen]

Thanks for the help! I'll run the CI and merge :)

[Jolanrensen]

@Kopilov Looks like the test org.jetbrains.kotlinx.dataframe.io.ArrowKtTest.testReadingAllTypesAsEstimatedNotNullableWithNulls now fails:

org.junit.ComparisonFailure: expected:<kotlin.Nothing?> but was:<kotlin.Nothing> expected:<kotlin.Nothing[?]> but was:<kotlin.Nothing[]>
  at org.jetbrains.kotlinx.dataframe.io.ExampleEstimatesAssertionsKt.assertEstimations(exampleEstimatesAssertions.kt:163)
  at org.jetbrains.kotlinx.dataframe.io.ArrowKtTest.testReadingAllTypesAsEstimatedNotNullableWithNulls(ArrowKtTest.kt:221)

This probably means we need my entire solution with NullabilityOptions after all... The tests are fine if I use this:

...

@JvmName("withTypeNullableNothingList")
private fun List<Nothing?>.withTypeNullable(
    expectedNulls: Boolean,
    nullabilityOptions: NullabilityOptions,
): Pair<List<Nothing?>, KType> {
    val nullable = nullabilityOptions.applyNullability(this, expectedNulls)
    return this to nothingType(nullable)
}

and then

is NullVector -> vector.values(range).withTypeNullable(field.isNullable, nullability)

[Kopilov]

@Jolanrensen applied, thanks