crazy timestamp bug in feather?

[randomgambit]

Hello the dream team,

Thanks for this wonderful package. I was playing with feather and some timestamps and I noticed some dangerous behavior. Maybe it is a bug.

Consider this

import pandas as pd
import feather
import numpy as np


df = pd.DataFrame({'string_time_utc' : [pd.to_datetime('2018-02-01 14:00:00.531'), 
                                  pd.to_datetime('2018-02-01 14:01:00.456'),
                                  pd.to_datetime('2018-03-05 14:01:02.200')]})

df['timestamp_est'] = pd.to_datetime(df.string_time_utc).dt.tz_localize('UTC').dt.tz_convert('US/Eastern').dt.tz_localize(None)

df
Out[17]: 
          string_time_utc           timestamp_est
0 2018-02-01 14:00:00.531 2018-02-01 09:00:00.531
1 2018-02-01 14:01:00.456 2018-02-01 09:01:00.456
2 2018-03-05 14:01:02.200 2018-03-05 09:01:02.200

Here I create the corresponding EST timestamp of my original timestamps (in UTC time).

Now saving the dataframe to csv or to feather will generate two completely different results.

df.to_csv('P://testing.csv')
df.to_feather('P://testing.feather')

Switching to R.

Using the good old csv gives me something a bit annoying, but expected. R thinks my timezone is UTC by default, and wrongly attached this timezone to timestamp_est. No big deal, I can always use with_tz or even better: import as character and process as timestamp while in R.

> dataframe <- read_csv('P://testing.csv')
Parsed with column specification:
cols(
  X1 = col_integer(),
  string_time_utc = col_datetime(format = ""),
  timestamp_est = col_datetime(format = "")
)
Warning message:
Missing column names filled in: 'X1' [1] 
> 
> dataframe %>% mutate(mytimezone = tz(timestamp_est))
# A tibble: 3 x 4
     X1 string_time_utc         timestamp_est          
  <int> <dttm>                  <dttm>                 
1     0 2018-02-01 14:00:00.530 2018-02-01 09:00:00.530
2     1 2018-02-01 14:01:00.456 2018-02-01 09:01:00.456
3     2 2018-03-05 14:01:02.200 2018-03-05 09:01:02.200
  mytimezone
  <chr>     
1 UTC       
2 UTC       
3 UTC 

Now look at what happens with feather:

> dataframe <- read_feather('P://testing.feather')
> 
> dataframe %>% mutate(mytimezone = tz(timestamp_est))
# A tibble: 3 x 3
  string_time_utc         timestamp_est           mytimezone
  <dttm>                  <dttm>                  <chr>     
1 2018-02-01 09:00:00.531 2018-02-01 04:00:00.531 ""        
2 2018-02-01 09:01:00.456 2018-02-01 04:01:00.456 ""        
3 2018-03-05 09:01:02.200 2018-03-05 04:01:02.200 ""     

My timestamps have been converted!!! pure insanity.
Am I missing something here?

Thanks!!

[wesm]

Can you report in Apache Arrow? Someone will need to investigate once we get R fully moved over from this codebase

[randomgambit]

hi wes you mean on the JIRA? can you send me the link? thx!

[wesm]

https://issues.apache.org/jira/browse/ARROW

[randomgambit]

done

[wesm]

Thanks. It will need a more descriptive / objective title

[randomgambit]

like "INSANE timestamp ERROR - win 400$ click here" I guess

[wesm]

We have hundreds of issues. We need to be able to understand the nature of the problem from the title

[randomgambit]

sure i was kidding :)

[randomgambit]

monsieur @wesm , do we have an update about this bug? timestamps are always tricky... thanks!!

[wesm]

I'm not sure the status, what does the JIRA issue you created say?

[randomgambit]

I dont think so ... :( this is sad

https://issues.apache.org/jira/browse/ARROW-3543?jql=project%20%3D%20ARROW%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22olaf%22%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

[randomgambit]

what do you thik, @wesm ? when is .13 being released? Thanks!

[wesm]

Please comment on ARROW-3543. I estimate timeline for 0.13 to be end of March