Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand Arrow reading support #129

Merged
merged 8 commits into from
Jul 18, 2022
Merged

Expand Arrow reading support #129

merged 8 commits into from
Jul 18, 2022

Conversation

Kopilov
Copy link
Contributor

@Kopilov Kopilov commented Jul 15, 2022

Arrow reading test with more data examples (generation is presented at https://github.com/Kopilov/arrow_example).

Read UInt-s without overflow, read date and time as date and time.

NullabilityOptions pattern.

@Kopilov Kopilov mentioned this pull request Jul 15, 2022
@Kopilov Kopilov changed the title Expanding Arrow reading support Expand Arrow reading support Jul 15, 2022
@koperagen
Copy link
Collaborator

This test fails. Can you check? We use it as a sample for docs
https://github.com/Kotlin/dataframe/blob/master/tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/Read.kt#L79

Also, in public class ArrowFeather : SupportedFormat readDataFrame functions should set NullabilityOptions.Widening as an argument

@Kopilov
Copy link
Contributor Author

Kopilov commented Jul 18, 2022

This test fails. Can you check? We use it as a sample for docs
https://github.com/Kotlin/dataframe/blob/master/tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/Read.kt#L79

This is because I removed file ./tests/src/test/resources/data-arrow_2.0.0_uncompressed.feather (that was originally copied to ./dataframe-arrow/src/test/resources/data-arrow_2.0.0_uncompressed.feather). Is using only one of them possible or we can only restore the copy?

@Kopilov
Copy link
Contributor Author

Kopilov commented Jul 18, 2022

Also, in public class ArrowFeather : SupportedFormat readDataFrame functions should set NullabilityOptions.Widening as an argument

Do I understand correctly that we should use Widening in SupportedFormat readDataFrame and Infer (formerly Keeping) as default in DataFrame.Companion.readArrowFeather methods?

@koperagen
Copy link
Collaborator

Also, in public class ArrowFeather : SupportedFormat readDataFrame functions should set NullabilityOptions.Widening as an argument

Do I understand correctly that we should use Widening in SupportedFormat readDataFrame and Infer (formerly Keeping) as default in DataFrame.Companion.readArrowFeather methods?

Yes

@koperagen
Copy link
Collaborator

This test fails. Can you check? We use it as a sample for docs
https://github.com/Kotlin/dataframe/blob/master/tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/Read.kt#L79

This is because I removed file ./tests/src/test/resources/data-arrow_2.0.0_uncompressed.feather (that was originally copied to ./dataframe-arrow/src/test/resources/data-arrow_2.0.0_uncompressed.feather). Is using only one of them possible or we can only restore the copy?

I think a copy won't harm

@koperagen koperagen merged commit a746109 into Kotlin:master Jul 18, 2022
@koperagen
Copy link
Collaborator

@Kopilov Thank you!

@Kopilov
Copy link
Contributor Author

Kopilov commented Jul 19, 2022

@koperagen you are welcome!

Our next step is to make saving :-)

@Kopilov Kopilov deleted the arrow branch July 19, 2022 06:12
@Jolanrensen Jolanrensen added this to the 0.9.0 milestone Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants