Skip to content

Conversation

@ggodik
Copy link
Contributor

@ggodik ggodik commented Aug 2, 2022

Newly supported types

  • Date32
  • Date64
  • Timestamp

csv.Reader currently supports Timestamps. Not adding Date32/64 support to CSV as the default behavior will stay the same and parse as the broadest timestamp type

https://issues.apache.org/jira/browse/ARROW-17273

@github-actions
Copy link

github-actions bot commented Aug 2, 2022

@github-actions
Copy link

github-actions bot commented Aug 2, 2022

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

t := w.schema.Field(j).Type.(*arrow.TimestampType)
for i := 0; i < arr.Len(); i++ {
if arr.IsValid(i) {
recs[i][j] = arr.Value(i).ToTime(t.Unit).Format("2006-01-02 15:04:05.999999999")
Copy link
Contributor Author

@ggodik ggodik Aug 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally I'd have TimestampFromStringInLocation but in reverse or FormattedString() like on Date32 or Date64

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can juts do arr.Value(i).ToTime(t.Unit).In(location).Format(.....) That's why I opted to have it use ToTime, because there's an In method on time.Time to easily shift it to whichever timezone location you need.

@zeroshade zeroshade merged commit 901e132 into apache:master Aug 2, 2022
@ggodik ggodik deleted the gg/csv branch August 2, 2022 17:30
@ursabot
Copy link

ursabot commented Aug 3, 2022

Benchmark runs are scheduled for baseline = dd96278 and contender = 901e132. 901e132 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.1% ⬆️0.07%] test-mac-arm
[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.32% ⬆️0.14%] ursa-thinkcentre-m75q
Buildkite builds:
[Failed] 901e132b ec2-t3-xlarge-us-east-2
[Failed] 901e132b test-mac-arm
[Finished] 901e132b ursa-i9-9960x
[Finished] 901e132b ursa-thinkcentre-m75q
[Failed] dd962782 ec2-t3-xlarge-us-east-2
[Finished] dd962782 test-mac-arm
[Finished] dd962782 ursa-i9-9960x
[Finished] dd962782 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants