Arrow writing #162
Arrow writing #162
Conversation
| citiesDeserialized["is_capital"] shouldBe citiesExampleFrame["is_capital"] | ||
| citiesDeserialized["population"] shouldBe citiesExampleFrame["population"] | ||
| citiesDeserialized["area"] shouldBe citiesExampleFrame["area"] | ||
| citiesDeserialized["settled"].type() shouldBe typeOf<LocalDate>() // cities["settled"].type() refers to FlexibleTypeImpl(LocalDate..LocalDate?) and does not match typeOf<LocalDate>() |
There was a problem hiding this comment.
Nice catch, i think it should be fixed in readArrow later as well
There was a problem hiding this comment.
FlexibleTypeImpl comes not from Arrow but from listOf(LocalDate.of()) in the sample generation.
What exactly do you want to fix?
dataframe-arrow/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/ArrowWriter.kt
Show resolved
Hide resolved
|
Wow, hi! That's a lot of work, thank you for the PR. I'll leave a couple of comments about the details, but right now i seem to lack general understanding, so i'll review it again a bit later. What do you think about making possible warnings a sealed class with overloaded toString for default behaviour? So that you can access actual exception in conversion or DataColumn and KType, etc. |
|
Sorry, sorry :D I just kept this open while merging other PR and accidentally merged this. Can you re-open? |
|
It seems that this can not be reopened. Duplicated in #169 by cherry-picking the branch manually. |
Saving DataFrame in Arrow format, as is or per schema.
Warnings are muted by default, this may be changed after #147 resolving.