fix: Don't cast all types to string for CSV #498
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
candiduslynx
requested review from
maaarcelino and
disq
and removed request for
a team and
maaarcelino
candiduslynx
force-pushed
the
fix/csv/supported-types
branch
from
51c9c6f
to
3f47c1f
Compare
candiduslynx
commented
May 9, 2024
| return dt | ||
| } | ||
| switch dt := dt.(type) { | ||
| case *arrow.MapType: |
There was a problem hiding this comment.
maps aren't supported
candiduslynx
commented
May 9, 2024
| } | ||
| switch dt := dt.(type) { | ||
| case *arrow.MapType: | ||
| case *arrow.FixedSizeListType: |
There was a problem hiding this comment.
lists are converted as list(converted elem)
disq
approved these changes
May 9, 2024
|
Can we wait for someone to complain about the CSV format before fixing this? We recently did a breaking change in all our destinations that are using this module due to #494 (which was justified as it fixed some stuff in Parquet, JSON and CSV for timestamps). Not sure we should do another one only because of a CSV issue nobody reported on. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes https://github.com/cloudquery/cloudquery-issues/issues/1644
Requires apache/arrow#41595 & bump to arrow v16.1.0.
Diff in files generated:
2021-01-02 00:00:00is now2021-01-02 00:00:00Z(so this can also be considered a follow-up for fix(deps): Upgradegithub.com/apache/arrow/gotov16#494)ValueStrignores null value set to"")