New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unwanted conversion from timedelta to float (but not datetime) #18493

Closed
dcolascione opened this Issue Nov 25, 2017 · 7 comments

Comments

Projects
None yet
4 participants
@dcolascione

dcolascione commented Nov 25, 2017

In the code below, masked assignment of a timedelta series converts the series values to float64. The same code running against a datetime correctly keeps the series dtype.

In [1]: import pandas as pd

In [2]: dseries=pd.Series(pd.date_range('1/1/2011', periods=5, freq='H')); dseries=dseries-dseries[0]; dseries
Out[2]: 
0   00:00:00
1   01:00:00
2   02:00:00
3   03:00:00
4   04:00:00
dtype: timedelta64[ns]

In [3]: dseries[dseries==dseries[0]]=None; dseries
Out[3]: 
0   -9.223372e+18
1    3.600000e+12
2    7.200000e+12
3    1.080000e+13
4    1.440000e+13
dtype: float64

In [4]: dseries=pd.Series(pd.date_range('1/1/2011', periods=5, freq='H')); dseries=dseries; dseries
Out[4]: 
0   2011-01-01 00:00:00
1   2011-01-01 01:00:00
2   2011-01-01 02:00:00
3   2011-01-01 03:00:00
4   2011-01-01 04:00:00
dtype: datetime64[ns]

In [5]: dseries[dseries==dseries[0]]=None; dseries
Out[5]: 
0                   NaT
1   2011-01-01 01:00:00
2   2011-01-01 02:00:00
3   2011-01-01 03:00:00
4   2011-01-01 04:00:00
dtype: datetime64[ns]
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 26, 2017

Contributor

show your version of pandas.

Contributor

jreback commented Nov 26, 2017

show your version of pandas.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 26, 2017

Contributor

this was broken in master as well.

This will patch, can you do a PR with appropriate tests?

(pandas) bash-3.2$ git diff
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index e537cb2..7a5877d 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -1956,7 +1956,8 @@ class TimeDeltaBlock(DatetimeLikeBlockMixin, IntBlock):
         tipo = maybe_infer_dtype_type(element)
         if tipo is not None:
             return issubclass(tipo.type, np.timedelta64)
-        return isinstance(element, (timedelta, np.timedelta64))
+        return is_integer(element) or isinstance(
+            element, (timedelta, np.timedelta64))
 
     def fillna(self, value, **kwargs):
Contributor

jreback commented Nov 26, 2017

this was broken in master as well.

This will patch, can you do a PR with appropriate tests?

(pandas) bash-3.2$ git diff
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index e537cb2..7a5877d 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -1956,7 +1956,8 @@ class TimeDeltaBlock(DatetimeLikeBlockMixin, IntBlock):
         tipo = maybe_infer_dtype_type(element)
         if tipo is not None:
             return issubclass(tipo.type, np.timedelta64)
-        return isinstance(element, (timedelta, np.timedelta64))
+        return is_integer(element) or isinstance(
+            element, (timedelta, np.timedelta64))
 
     def fillna(self, value, **kwargs):
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 26, 2017

Contributor

tests can go
pandas/tests/indexing/test_timedelta.py
add tests setting with each of [None, np.nan, pd.NaT]

Contributor

jreback commented Nov 26, 2017

tests can go
pandas/tests/indexing/test_timedelta.py
add tests setting with each of [None, np.nan, pd.NaT]

@dcolascione

This comment has been minimized.

Show comment
Hide comment
@dcolascione

dcolascione Nov 26, 2017

Thanks for the fast response! I'm not sure I'll be able to get a dev environment up and running soon. There are various versioning conflicts with Cython that make this more than a five minute job. :-(

dcolascione commented Nov 26, 2017

Thanks for the fast response! I'm not sure I'll be able to get a dev environment up and running soon. There are various versioning conflicts with Cython that make this more than a five minute job. :-(

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 26, 2017

Contributor

There are various versioning conflicts with Cython that make this more than a five minute job. :-(

really?

http://pandas.pydata.org/pandas-docs/stable/contributing.html#

(pandas) bash-3.2$ conda create -n foo --file ci/requirements_dev.txt  --dry
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /Users/jreback/miniconda3/envs/foo:

The following NEW packages will be INSTALLED:

    alabaster:                0.7.10-py35hb692fe1_0    defaults   
    asn1crypto:               0.22.0-py35ha010153_1    defaults   
    babel:                    2.5.0-py35h6f05a75_0     defaults   
    boto:                     2.48.0-py35h10f7326_1    defaults   
    boto3:                    1.4.7-py35hd462e4d_0     defaults   
    botocore:                 1.7.20-py35h01c6a2e_0    defaults   
    ca-certificates:          2017.08.26-ha1e5d58_0    defaults   
    certifi:                  2017.11.5-py35hd00889a_0 defaults   
    cffi:                     1.10.0-py35hca2225f_1    defaults   
    chardet:                  3.0.4-py35h16a84c2_1     defaults   
    click:                    6.7-py35hcc65ea6_0       defaults   
    cookies:                  2.2.1-py35_0             defaults   
    cryptography:             2.0.3-py35hde935b2_1     defaults   
....
Contributor

jreback commented Nov 26, 2017

There are various versioning conflicts with Cython that make this more than a five minute job. :-(

really?

http://pandas.pydata.org/pandas-docs/stable/contributing.html#

(pandas) bash-3.2$ conda create -n foo --file ci/requirements_dev.txt  --dry
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /Users/jreback/miniconda3/envs/foo:

The following NEW packages will be INSTALLED:

    alabaster:                0.7.10-py35hb692fe1_0    defaults   
    asn1crypto:               0.22.0-py35ha010153_1    defaults   
    babel:                    2.5.0-py35h6f05a75_0     defaults   
    boto:                     2.48.0-py35h10f7326_1    defaults   
    boto3:                    1.4.7-py35hd462e4d_0     defaults   
    botocore:                 1.7.20-py35h01c6a2e_0    defaults   
    ca-certificates:          2017.08.26-ha1e5d58_0    defaults   
    certifi:                  2017.11.5-py35hd00889a_0 defaults   
    cffi:                     1.10.0-py35hca2225f_1    defaults   
    chardet:                  3.0.4-py35h16a84c2_1     defaults   
    click:                    6.7-py35hcc65ea6_0       defaults   
    cookies:                  2.2.1-py35_0             defaults   
    cryptography:             2.0.3-py35hde935b2_1     defaults   
....
@zinklandi

This comment has been minimized.

Show comment
Hide comment
@zinklandi

zinklandi Nov 30, 2017

Hey guys!

I think the selection and the comparison which is done like ...

dseries[dseries==dseries[0]]

... produces a unwanted transformation of the series dtype. A good way to transform the series back is the from float or int is the following way:

pd.to_timedelta(dseries[dseries < '30 sec'])

I'm not sure if the transformation is a bug, but I hope that I could help :)

This is also something I found:
screen shot 2017-11-30 at 08 44 23

I think there's a transformation in the series which transforms a pandas.timedelta into numpy.timedelta..

zinklandi commented Nov 30, 2017

Hey guys!

I think the selection and the comparison which is done like ...

dseries[dseries==dseries[0]]

... produces a unwanted transformation of the series dtype. A good way to transform the series back is the from float or int is the following way:

pd.to_timedelta(dseries[dseries < '30 sec'])

I'm not sure if the transformation is a bug, but I hope that I could help :)

This is also something I found:
screen shot 2017-11-30 at 08 44 23

I think there's a transformation in the series which transforms a pandas.timedelta into numpy.timedelta..

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 30, 2017

Contributor

@SDzinklandi the patch is above if you want to submit a PR

Contributor

jreback commented Nov 30, 2017

@SDzinklandi the patch is above if you want to submit a PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment