detect date fields in Apache Arrow tables #263
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mootari
reviewed
May 22, 2024
src/table.js
Outdated
| @@ -355,6 +355,8 @@ function alignof(base = {}, data, columns) { | |||
| } | |||
|
|
|||
| function type(data, column) { | |||
| // duck-type Arrow table date fields | |||
| if (String(data?.schema?.fields?.find?.(d => d?.name === column)).match(/<MILLISECOND>/)) return "date"; | |||
There was a problem hiding this comment.
Maybe a better fit:
Suggested change
| if (String(data?.schema?.fields?.find?.(d => d?.name === column)).match(/<MILLISECOND>/)) return "date"; | |
| if (String(data?.schema?.fields?.find?.(d => d?.name === column)).endsWith("<MILLISECOND>") return "date"; |
I also wonder if the string representation is part of arrow's public API. Should we perhaps check against the typeId instead?
Edit: sorry, accidentally dropped a )
There was a problem hiding this comment.
I don't know… maybe… I tried to identify something that had a chance to be a bit stable, but it's a moving target.
There was a problem hiding this comment.
From what I recall the positive typeIds are public, and negative IDs are meant for internal use.
Edit: sort of
* **Note**: Only enum values 0-17 (NONE through Map) are written to an Arrow
* IPC payload.
Co-authored-by: Fabian Iwand <mootari@users.noreply.github.com>
Fil
commented
May 22, 2024
2 tasks
Fil
added a commit
to observablehq/plot
that referenced
this pull request
Jun 21, 2024
uses the same duck type test as observablehq/inputs#263 closes observablehq/framework#1376
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
addresses the Inputs.table part of observablehq/framework#1376
I thought we could extend it to duck-type other field types (numbers vs strings), but there is no real need beyond dates and it would add complexity.