ARROW-315: finalize timestamp #156

julienledem · 2016-10-03T23:26:53Z

No description provided.

wesm · 2016-10-04T03:24:37Z

format/Message.fbs

@@ -64,8 +64,11 @@ table Date {
 table Time {
 }

+enum TimeUnit: short { MILLISECOND, MICROSECOND }


Can we add nanosecond? This is quite common in the financial world for example (part of the reason that pandas uses nanoseconds)

Also, Unix timestamps are seconds since the epoch. Not sure how commonly occurring these are in practice

+1 for adding both, SECOND and NANOSECOND

I added millis and micros because those are the ones that Parquet supports.
For now nanos and second metadata would be lost in saving to Parquet and stored as int?

https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#datetime-types

That's fine to drop the metadata in those cases when writing to Parquet. You would need to write key-value metadata to be able to recover the exact metadata (timestamp in nanos)

wesm · 2016-10-04T03:25:02Z

format/Message.fbs

 table Timestamp {
-  timezone: string;


Is the thought that a time zone (if any) would go in the custom metadata?

Timezoned data should be converted to UTC before putting in arrow.
That's simpler.
We can add timezone later on if necessary.

That's fine with me. My experience has been that having both time zone naive and non-naive timestamps is more trouble than it's worth

julienledem · 2016-10-04T21:38:01Z

@wesm @xhochy : I have added SECOND and NANOSECOND
good to go?

wesm · 2016-10-04T21:41:11Z

+1, thank you

@xhochy

I wrote a sample file and the metadata seems to be correct. @xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#156 from majetideepak/PARQUET-711 and squashes the following commits: 25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707 8b4784d [Deepak Majeti] Review comments to add methods back fdbc761 [Deepak Majeti] fix clang error and comments c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata ada3ac2 [Deepak Majeti] clang format d9c9131 [Deepak Majeti] Use metadata builders in parquet writer

@xhochy

I wrote a sample file and the metadata seems to be correct. @xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#156 from majetideepak/PARQUET-711 and squashes the following commits: 25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707 8b4784d [Deepak Majeti] Review comments to add methods back fdbc761 [Deepak Majeti] fix clang error and comments c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata ada3ac2 [Deepak Majeti] clang format d9c9131 [Deepak Majeti] Use metadata builders in parquet writer Change-Id: Idaef448d5a5ac4b29794f141d2b5c87a1e4f40d5

@xhochy

I wrote a sample file and the metadata seems to be correct. @xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#156 from majetideepak/PARQUET-711 and squashes the following commits: 25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707 8b4784d [Deepak Majeti] Review comments to add methods back fdbc761 [Deepak Majeti] fix clang error and comments c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata ada3ac2 [Deepak Majeti] clang format d9c9131 [Deepak Majeti] Use metadata builders in parquet writer Change-Id: Idaef448d5a5ac4b29794f141d2b5c87a1e4f40d5

@xhochy

I wrote a sample file and the metadata seems to be correct. @xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#156 from majetideepak/PARQUET-711 and squashes the following commits: 25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707 8b4784d [Deepak Majeti] Review comments to add methods back fdbc761 [Deepak Majeti] fix clang error and comments c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata ada3ac2 [Deepak Majeti] clang format d9c9131 [Deepak Majeti] Use metadata builders in parquet writer Change-Id: Idaef448d5a5ac4b29794f141d2b5c87a1e4f40d5

@xhochy

I wrote a sample file and the metadata seems to be correct. @xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#156 from majetideepak/PARQUET-711 and squashes the following commits: 25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707 8b4784d [Deepak Majeti] Review comments to add methods back fdbc761 [Deepak Majeti] fix clang error and comments c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata ada3ac2 [Deepak Majeti] clang format d9c9131 [Deepak Majeti] Use metadata builders in parquet writer Change-Id: Idaef448d5a5ac4b29794f141d2b5c87a1e4f40d5

@xhochy

I wrote a sample file and the metadata seems to be correct. @xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#156 from majetideepak/PARQUET-711 and squashes the following commits: 25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707 8b4784d [Deepak Majeti] Review comments to add methods back fdbc761 [Deepak Majeti] fix clang error and comments c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata ada3ac2 [Deepak Majeti] clang format d9c9131 [Deepak Majeti] Use metadata builders in parquet writer Change-Id: Iabe80b1cbe3fd8f1de6239187058b6402b160975

@xhochy

I wrote a sample file and the metadata seems to be correct. @xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#156 from majetideepak/PARQUET-711 and squashes the following commits: 25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707 8b4784d [Deepak Majeti] Review comments to add methods back fdbc761 [Deepak Majeti] fix clang error and comments c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata ada3ac2 [Deepak Majeti] clang format d9c9131 [Deepak Majeti] Use metadata builders in parquet writer Change-Id: Iabe80b1cbe3fd8f1de6239187058b6402b160975

@xhochy

I wrote a sample file and the metadata seems to be correct. @xhochy I fixed some missing metadata like `dictionary_page_offset`. You might want to check if this fixes the Drill problem. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes #156 from majetideepak/PARQUET-711 and squashes the following commits: 25f5a7e [Deepak Majeti] fix schema and descr. Resolves PARQUET-705 and PARQUET-707 8b4784d [Deepak Majeti] Review comments to add methods back fdbc761 [Deepak Majeti] fix clang error and comments c6cb071 [Deepak Majeti] convert DCHECKS to Exceptions in metadata ada3ac2 [Deepak Majeti] clang format d9c9131 [Deepak Majeti] Use metadata builders in parquet writer Change-Id: Iabe80b1cbe3fd8f1de6239187058b6402b160975

…t-get-sql-info [Flight SQL CPP] Add support for command `GetSqlInfo` on client side

ARROW-315: finalize timestamp

86cae98

wesm reviewed Oct 4, 2016

View reviewed changes

review feedback

0ee017f

asfgit closed this in 7fb4d24 Oct 4, 2016

julienledem deleted the timestamp branch April 25, 2017 18:31

rafael-telles pushed a commit to rafael-telles/arrow that referenced this pull request Oct 19, 2021

Merge pull request apache#156 from rafael-telles/flight-sql-cpp-clien…

3716397

…t-get-sql-info [Flight SQL CPP] Add support for command `GetSqlInfo` on client side

asfimport mentioned this pull request May 24, 2017

Finalize timestamp type #15760

Closed

paleolimbot mentioned this pull request Jan 28, 2023

[R] Crash on MacOS (x86) when running tests with homebrew apache-arrow also installed #33903

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-315: finalize timestamp #156

ARROW-315: finalize timestamp #156

julienledem commented Oct 3, 2016

wesm Oct 4, 2016

wesm Oct 4, 2016

xhochy Oct 4, 2016

julienledem Oct 4, 2016

julienledem Oct 4, 2016

wesm Oct 4, 2016

wesm Oct 4, 2016

julienledem Oct 4, 2016

wesm Oct 4, 2016

julienledem commented Oct 4, 2016

wesm commented Oct 4, 2016

ARROW-315: finalize timestamp #156

ARROW-315: finalize timestamp #156

Conversation

julienledem commented Oct 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julienledem commented Oct 4, 2016

wesm commented Oct 4, 2016