Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Timestamp.replace chaining not compat with datetime.replace #15683

Closed
sscherfke opened this issue Mar 14, 2017 · 29 comments

Comments

Projects
None yet
2 participants
@sscherfke
Copy link

commented Mar 14, 2017

Code Sample, a copy-pastable example if possible

import pytz
import pandas as pd
from datetime import datetime
pytz.timezone('CET').localize(datetime(2016, 3, 27, 1), is_dst=None)
pytz.timezone('CET').localize(pd.Timestamp(datetime(2016, 3, 27, 1)), is_dst=None)

Problem description

The above code runs with Pandas 0.18 but raises the following exception with Pandas 0.19:

>>> pytz.timezone('CET').localize(pd.Timestamp(datetime(2016, 3, 27, 1)), is_dst=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/localhome/stefan/emsconda/envs/popeye/lib/python3.6/site-packages/pytz/tzinfo.py", line 327, in localize
    raise NonExistentTimeError(dt)
pytz.exceptions.NonExistentTimeError: 2016-03-27 01:00:00

Is this an intentional API breakage of 0.19 or a bug?

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.12-100.fc24.x86_64+debug
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: de_DE.UTF-8

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 34.3.0
Cython: None
numpy: 1.12.0
scipy: 0.18.1
statsmodels: None
xarray: None
IPython: 5.3.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2016.10
blosc: 1.5.0
bottleneck: None
tables: None
numexpr: None
matplotlib: 2.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: None
boto: None
pandas_datareader: None

@jreback

This comment has been minimized.

Copy link
Contributor

commented Mar 14, 2017

This is correct according to pytz doc-string.

In [8]: pytz.timezone('CET').localize(Timestamp(datetime(2016, 3, 27, 1)), is_dst=True)
Out[8]: Timestamp('2016-03-27 00:00:00+0100', tz='CET')

In [9]: pytz.timezone('CET').localize(Timestamp(datetime(2016, 3, 27, 1)), is_dst=False)
Out[9]: Timestamp('2016-03-27 01:00:00+0100', tz='CET')

In [10]: pytz.timezone('CET').localize(Timestamp(datetime(2016, 3, 27, 1)), is_dst=None)
---------------------------------------------------------------------------
NonExistentTimeError                      Traceback (most recent call last)
<ipython-input-10-6cbd34e0bbef> in <module>()
----> 1 pytz.timezone('CET').localize(Timestamp(datetime(2016, 3, 27, 1)), is_dst=None)

/Users/jreback/miniconda3/envs/pandas/lib/python3.5/site-packages/pytz/tzinfo.py in localize(self, dt, is_dst)
    325             # If we refuse to guess, raise an exception.
    326             if is_dst is None:
--> 327                 raise NonExistentTimeError(dt)
    328 
    329             # If we are forcing the pre-DST side of the DST transition, we

NonExistentTimeError: 2016-03-27 01:00:00

@jreback jreback added the Timezones label Mar 14, 2017

@jreback

This comment has been minimized.

Copy link
Contributor

commented Mar 14, 2017

actually I find the pytz behavior of is_dst=None to be just odd. They are conflating too many things into a single argument I am afraid.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Mar 15, 2017

Okay, thx for the feedback.

@sscherfke sscherfke closed this Mar 15, 2017

@sscherfke

This comment has been minimized.

Copy link
Author

commented Mar 15, 2017

We did some more research on this issue and found the following:

The problem occurs during DST-changes when we (de)normalize input dates from dates with tzinfo to UTC dates and back from tz-less UTC dates to dates with a tzinfo.

The stdlib docs states:

“Return a date with the same value, except for those parameters given new values by whichever keyword arguments are specified.”

Lets test this:

import pytz
import pandas as pd
from datetime import datetime

# Base datetime and a tzinfo object
dt = datetime(2016, 3, 27, 1)
tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo

# Expected: tzinfo replaced, actual date value unchanged:
print('Datetimes:')
print(dt.replace(tzinfo=tzinfo))
print(dt.replace(tzinfo=tzinfo).replace(tzinfo=None))

# Unexpected behaviour in pandas 0.19.x:
# Other values than tzinfo were changed:
print('Pandas Timestamp:')
print(pd.Timestamp(dt).replace(tzinfo=tzinfo))
print(pd.Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None))

Pandas 0.18.1:

Datetimes:
2016-03-27 01:00:00+01:00
2016-03-27 01:00:00
Pandas Timestamp:
2016-03-27 01:00:00+01:00
2016-03-27 01:00:00        # ok

Pandas 0.19.2:

Datetimes:
2016-03-27 01:00:00+01:00
2016-03-27 01:00:00
Pandas Timestamp:
2016-03-27 01:00:00+01:00
2016-03-27 00:00:00        # unexpected

The datetime in the last row of the Pandas 0.19.2 output is incorrect.

This readily occurs in the context of pytz as localize() and normalize() do that all the time (here: pytz 2016.10) in pytz.tzinfo.DstTzInfo.localize, line 314, loc_dt = tzinfo.normalize(dt.replace(tzinfo=tzinfo)) is executed and in pytz.tzinfo.DstTzInfo.normalize, line 239, dt = dt.replace(tzinfo=None) is executed.

Interestingly, the issue only occurs if we do the “double replace” but not if we directly initialize a datetime with a tzinfo:

print(pd.Timestamp(datetime(2016, 3, 27, 1, tzinfo=tzinfo))
print(pd.Timestamp(datetime(2016, 3, 27, 1, tzinfo=tzinfo).replace(tzinfo=None))

Pandas 0.18.1:

2016-03-27 01:00:00+01:00
2016-03-27 01:00:00

Pandas 0.19.2:

2016-03-27 01:00:00+01:00
2016-03-27 01:00:00

I looked at the change logs and coudn’t find anything related to this issue.

@sscherfke sscherfke reopened this Mar 15, 2017

@jreback

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2017

I guess this is a bug, Timestamp.replace should act exactly like datetime.replace. It is overriden because it needs to handle parameter validation and nanoseconds. So [21] should match [19]

In [18]: dt.replace(tzinfo=tzinfo)
Out[18]: datetime.datetime(2016, 3, 27, 1, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)

In [19]: dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
Out[19]: datetime.datetime(2016, 3, 27, 1, 0)

In [20]: pd.Timestamp(dt).replace(tzinfo=tzinfo)
Out[20]: Timestamp('2016-03-27 01:00:00+0100', tz='CET')

In [21]: pd.Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
Out[21]: Timestamp('2016-03-27 00:00:00')

All that said, I would never use .replace directly, and more naturally simply tz_localize and tz_convert. (including ambiguity over transitions and such).

In [25]: pd.Timestamp(dt).tz_localize(tzinfo)
Out[25]: Timestamp('2016-03-27 01:00:00+0100', tz='CET')

In [26]: pd.Timestamp(dt).tz_localize(tzinfo).tz_localize(None)
Out[26]: Timestamp('2016-03-27 01:00:00')

you are welcome to submit a PR to fix.

@jreback

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2017

f8bd08e is the change (has been modified slightly since then).

@jreback jreback added this to the Next Major Release milestone Mar 15, 2017

@jreback jreback changed the title Unexpected behavior b/w 0.18 and 0.19 when converting timezones with pytz BUG: Timestamp.replace chaining not compat with datetime.replace Mar 15, 2017

@sscherfke

This comment has been minimized.

Copy link
Author

commented Mar 29, 2017

Unfortunately, we currently don't have the time to get familiar with the pandas internals and fix this issue ourselves.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Mar 29, 2017

I added a regression test for this issues as follows:

diff --git a/pandas/tests/tseries/test_timezones.py b/pandas/tests/tseries/test_timezones.py
index 1fc0e1b..75d4872 100644
--- a/pandas/tests/tseries/test_timezones.py
+++ b/pandas/tests/tseries/test_timezones.py
@@ -1233,6 +1233,18 @@ class TestTimeZones(tm.TestCase):
             self.assertEqual(result_pytz.to_pydatetime().tzname(),
                              result_dateutil.to_pydatetime().tzname())

+        # issue 15683
+        dt = datetime(2016, 3, 27, 1)
+        tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo
+        # This should work:
+        result_dt = dt.replace(tzinfo=tzinfo)
+        result_pd = Timestamp(dt).replace(tzinfo=tzinfo)
+        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
+        # self.assertEqual(result_dt, result_pd.to_datetime())  # This fails!!!
+        # This should fail:
+        result_dt = dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
+        result_pd = Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
+        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
+        # self.assertEqual(result_dt, result_pd.to_datetime())
+
     def test_index_equals_with_tz(self):
         left = date_range('1/1/2011', periods=100, freq='H', tz='utc')
         right = date_range('1/1/2011', periods=100, freq='H', tz='US/Eastern')

Surprisingly, the assertEqual() using to_datetime() fails. I don't know if this is another issue or not:

>       self.assertEqual(result_dt, result_pd.to_datetime())
E       AssertionError: datetime.datetime(2016, 3, 27, 1, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>) != datetime.datetime(2016, 3, 27, 0, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)

I still have no Idea how to fix this.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Mar 31, 2017

I played around with with it a little bit more and something looks very broken:

>>> import datetime, pandas, pytz
>>>
>>> # Two equal datetimes:
>>> dt = datetime.datetime(2016, 3, 27, 1)
>>> pd = pandas.Timestamp(dt)
>>> dt == pd
True
>>> dt == pd.to_pydatetime()
True
>>> dt.timestamp() == pd.timestamp()
True
>>>
>>> # Let's introduce timezones and stuff breaks:
>>> 
>>> tzinfo = pytz.timezone('CET')
>>> rdt = dt.replace(tzinfo=tzinfo)
>>> rpd = pd.replace(tzinfo=tzinfo)
>>>
>>> rdt == rpd  # What?
False
>>> rdt == rpd.to_pydatetime()  # Really?
False
>>> rdt.timestamp() == rpd.timestamp()  # Why is this True now?
True
>>> # What do we have?
>>> rdt
datetime.datetime(2016, 3, 27, 1, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)
>>> rpd  # This *looks* like rdt but is *not equal* to it.
Timestamp('2016-03-27 01:00:00+0100', tz='CET')
>>> rpd.to_pydatetime()  # This is cleary not wanted:
datetime.datetime(2016, 3, 27, 0, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)
>>>
>>> # This seems to be the logical result of the above bug:
>>> ndt = rdt.replace(tzinfo=None)
>>> npd = rpd.replace(tzinfo=None)
>>> ndt
datetime.datetime(2016, 3, 27, 1, 0)
>>> npd
Timestamp('2016-03-27 00:00:00')
>>> npd.to_pydatetime()
datetime.datetime(2016, 3, 27, 0, 0)
>>> ndt == dt
True
>>> npd == pd
False
@sscherfke

This comment has been minimized.

Copy link
Author

commented Mar 31, 2017

The Timestamp constructor already seems to be broken:

>>> dttz = datetime.datetime(2016, 3, 27, 1, tzinfo=tzinfo)
>>> pdtz = pandas.Timestamp(2016, 3, 27, 1, tzinfo=tzinfo)
>>> dttz
datetime.datetime(2016, 3, 27, 1, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)
>>> pdtz  # Where is the tzinfo?
Timestamp('2016-03-27 01:00:00')
>>> dttz.timestamp() == pdtz.timestamp()  # Expected
True
>>> dttz == pdtz  # Unexpected
False
>>> dttz == pdtz.to_pydatetime()  # Unexpected
False
@jreback

This comment has been minimized.

Copy link
Contributor

commented Mar 31, 2017

@sscherfke datetime.datetime has a different underlying representation

In [1]: dt = pd.Timestamp('2016-03-27 01:00:00', tz='CET')

In [2]: dt
Out[2]: Timestamp('2016-03-27 00:00:00+0100', tz='CET')

In [3]: dt.tz_convert('UTC')
Out[3]: Timestamp('2016-03-26 23:00:00+0000', tz='UTC')

In [4]: dt.tz_convert('UTC').value
Out[4]: 1459033200000000000

In [5]: dt.value
Out[5]: 1459033200000000000

In [6]: dt.tz
Out[6]: <DstTzInfo 'CET' CET+1:00:00 STD>

In [7]: dt.tz_convert('UTC').tz
Out[7]: <UTC>

TImestamp keeps UTC time always and the tz as a parameter. This always efficient manipulation.

You are encourage to use tz_localize/tz_convert as these correctly manipulate all dst / tz's and work across different tz vendors.

the construction as a small issue xref in #15777

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 4, 2017

Okay, maybe my last example might then not be related to this issue. But the problem with replace(tzinfo) (it does not only replace the tzinfo but also alter the the actual date/time) remains.

I'd really like to help fixing this issue but Pandas has a very huge code base and I'm a very new Pandas user... :-/

@jreback

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2017

@sscherfke well .replace is actually a very straightforward method, though its in cython, and it does call other things.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 4, 2017

Yes, the other things is the problem. Finding out what they are supposed to do and what they actually to and which other thing actually it the culprit for this issue. :)

@jreback

This comment has been minimized.

Copy link
Contributor

commented Apr 8, 2017

now that #15934 is merged the construction issues should be fixed, FYI.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 19, 2017

Yes, the construction issues are fixed now.

I hoped that this might (accidentally) fix this issue, but it doesn't:

>>> import datetime, pandas, pytz
>>> tzinfo = pytz.timezone('CET')
>>> dt = datetime.datetime(2016, 3, 27, 1)
>>> pd = pandas.Timestamp(dt)
>>> dttz = dt.replace(tzinfo=tzinfo)
>>> pdtz1 = pd.replace(tzinfo=tzinfo)
>>> pdtz2 = pandas.Timestamp('2016-03-27 01:00', tz='CET')
>>> dttz == pdtz1
False
>>> dttz == pdtz2
True
>>> for x in [pdtz1, pdtz2]:
...     print(x, x.tzinfo, x.timestamp(), x.value, x.to_pydatetime())
...
2016-03-27 01:00:00+01:00 CET 1459036800.0 1459033200000000000 2016-03-27 00:00:00+01:00
2016-03-27 01:00:00+01:00 CET 1459036800.0 1459036800000000000 2016-03-27 01:00:00+01:00

As you can see, the value of both Timestamp differs, so I guess replace() breaks it somehow.

replace() (when called on a none-timezoned TS) calls the following four methods in this order:

  • pandas_datetime_to_datetimestruct()
  • pandas_datetimestruct_to_datetime()
  • tz_convert_single()
  • create_timestamp_from_ts()

The pandas_a_to_b() methods are neither defined nor imported in the module (??), so I took a closer look at the remaining two.

create_timestamp_from_ts() does not appear to do any calculations on the value.

So I think tz_convert_single() remains as the most probable culprit.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 19, 2017

Yes, tz_convert_single() is the culprit. I added a few prints in replace(). Before that method is called at the end, the value is 1459040400000000000 and afterwards it is 1459033200000000000. The difference is 2h (which is wrong – it should be 1h).

@jreback

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2017

yeah this should not be converting, instead it should be localizing.

diff --git a/pandas/_libs/tslib.pyx b/pandas/_libs/tslib.pyx
index c471d46..6356073 100644
--- a/pandas/_libs/tslib.pyx
+++ b/pandas/_libs/tslib.pyx
@@ -732,7 +732,9 @@ class Timestamp(_Timestamp):
 
         # set tz if needed
         if _tzinfo is not None:
-            value = tz_convert_single(value, _tzinfo, 'UTC')
+            value = tz_localize_to_utc(np.array([value], dtype='i8'), _tzinfo,
+                                       ambiguous='raise',
+                                       errors='raise')[0]
 
         result = create_timestamp_from_ts(value, dts, _tzinfo, self.freq)
         return result
@jreback

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2017

this breaks another test, but passes (so you would make this into an actual test)

In [1]: import datetime, pandas, pytz
   ...: tzinfo = pytz.timezone('CET')
   ...: dt = datetime.datetime(2016, 3, 27, 1)
   ...: pd = pandas.Timestamp(dt)
   ...: dttz = dt.replace(tzinfo=tzinfo)
   ...: pdtz1 = pd.replace(tzinfo=tzinfo)
   ...: pdtz2 = pandas.Timestamp('2016-03-27 01:00', tz='CET')
   ...: 

In [2]: dttz == pdtz1
Out[2]: True

In [3]: dttz == pdtz2
Out[3]: True
@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 19, 2017

I am very confused about what's happening inside Pandas:

>>> dt = datetime.datetime(2016, 3, 27, 1)
>>> datetime.datetime.fromtimestamp(pandas.Timestamp(dt).timestamp())
datetime.datetime(2016, 3, 27, 1, 0)
>>> datetime.datetime.fromtimestamp(pandas.Timestamp(dt).value / 1000000000)
datetime.datetime(2016, 3, 27, 3, 0)

I thought value would be a high-res UTC timestamp but it is actually two hours ahead of timestamp() (at least in this case).

When value is converted from CET to UTC at the end of replace(), tz_convert_single() detects that value is summer time (CEST) (because 2016-03-27 03:00 is CEST), it calculates an offset of 2h.

*edit: Saw you comments only after I wrote this comment.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 19, 2017

A test case like this will show the issue and pass when your proposed fix is applied:

    def test_issue_15683(self):
        # issue 15683
        dt = datetime(2016, 3, 27, 1)
        tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo
        result_dt = dt.replace(tzinfo=tzinfo)
        result_pd = Timestamp(dt).replace(tzinfo=tzinfo)
        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
        self.assertEqual(result_dt, result_pd.to_pydatetime())
        self.assertEqual(result_dt, result_pd)
        result_dt = dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
        result_pd = Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
        self.assertEqual(result_dt, result_pd.to_pydatetime())
        self.assertEqual(result_dt, result_pd)
@jreback

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2017

happy to take a PR to fix as I said.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 19, 2017

What about the breaking test?

@jreback

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2017

if you'd like to delve into that would be helpful

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 20, 2017

More problems (with our fix):

>>> import datetime, pandas, pytz
>>> tzinfo = pytz.timezone('CET')
>>>
>>> # Reference case with datetime.datetime object
>>> pd = pandas.Timestamp('2016-10-30 01:15').to_pydatetime()
>>> pd
datetime.datetime(2016, 10, 30, 1, 15)
>>> tzinfo.localize(pd, is_dst=True)
datetime.datetime(2016, 10, 30, 1, 15, tzinfo=<DstTzInfo 'CET' CEST+2:00:00 DST>)
>>>
>>> # Error in Pandas
>>> pd = pandas.Timestamp('2016-10-30 01:15')
>>> pd
Timestamp('2016-10-30 01:15:00')
>>> tzinfo.localize(pd, is_dst=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../envs/pandas/lib/python3.6/site-packages/pytz/tzinfo.py", line 314, in localize
    loc_dt = tzinfo.normalize(dt.replace(tzinfo=tzinfo))
  File ".../envs/pandas/lib/python3.6/site-packages/pytz/tzinfo.py", line 242, in normalize
    return self.fromutc(dt)
  File ".../envs/pandas/lib/python3.6/site-packages/pytz/tzinfo.py", line 187, in fromutc
    return (dt + inf[0]).replace(tzinfo=self._tzinfos[inf])
  File "pandas/_libs/tslib.pyx", line 735, in pandas._libs.tslib.Timestamp.replace (pandas/_libs/tslib.c:14931)
    value = tz_localize_to_utc(np.array([value], dtype='i8'), _tzinfo,
  File "pandas/_libs/tslib.pyx", line 4582, in pandas._libs.tslib.tz_localize_to_utc (pandas/_libs/tslib.c:77718)
    raise pytz.AmbiguousTimeError(
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from Timestamp('2016-10-30 02:15:00'), try using the 'ambigu
ous' argument

This is weired as 01:15 is actually not ambiguous (02:15 would be). I guess the problem arises because we convert from our destination tz to UTC in replace().

In the old version (without the fix) there would be no error but a wrong result (1h offset).

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 24, 2017

If a Timestamp has a tzinfo (e.g., UTC or CET), (Timestamp.value / 1_000_000_000) == pd.timestamp().

If a Timestamp does not have a tzinfo, (Timestamp.value / 1_000_000_000) - pd.timestamp() is the offset of my local timezone to UTC. Why is this by chance? What is this offset and why is it?

@jreback

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2017

@sscherfke not sure what you are asking.

@sscherfke

This comment has been minimized.

Copy link
Author

commented Apr 24, 2017

I finally found a solution that works. All tests in test_timezones are passing and our own code seems to work as well. :)

diff --git a/pandas/_libs/tslib.pyx b/pandas/_libs/tslib.pyx
index c471d46..c418059 100644
--- a/pandas/_libs/tslib.pyx
+++ b/pandas/_libs/tslib.pyx
@@ -685,14 +685,16 @@ class Timestamp(_Timestamp):
         cdef:
             pandas_datetimestruct dts
             int64_t value
-            object _tzinfo, result, k, v
+            object _tzinfo, result, k, v, ts_input
             _TSObject ts
 
         # set to naive if needed
         _tzinfo = self.tzinfo
         value = self.value
         if _tzinfo is not None:
-            value = tz_convert_single(value, 'UTC', _tzinfo)
+            value_tz = tz_convert_single(value, _tzinfo, 'UTC')
+            offset = value - value_tz
+            value += offset
 
         # setup components
         pandas_datetime_to_datetimestruct(value, PANDAS_FR_ns, &dts)
@@ -726,16 +728,14 @@ class Timestamp(_Timestamp):
             _tzinfo = tzinfo
 
         # reconstruct & check bounds
-        value = pandas_datetimestruct_to_datetime(PANDAS_FR_ns, &dts)
+        ts_input = datetime(dts.year, dts.month, dts.day, dts.hour, dts.min,
+                            dts.sec, dts.us, tzinfo=_tzinfo)
+        ts = convert_to_tsobject(ts_input, _tzinfo, None, 0, 0)
+        value = ts.value + (dts.ps // 1000)
         if value != NPY_NAT:
             _check_dts_bounds(&dts)
 
-        # set tz if needed
-        if _tzinfo is not None:
-            value = tz_convert_single(value, _tzinfo, 'UTC')
-
-        result = create_timestamp_from_ts(value, dts, _tzinfo, self.freq)
-        return result
+        return create_timestamp_from_ts(value, dts, _tzinfo, self.freq)
 
     def isoformat(self, sep='T'):
         base = super(_Timestamp, self).isoformat(sep=sep)
diff --git a/pandas/tests/tseries/test_timezones.py b/pandas/tests/tseries/test_timezones.py
index 06b6bbb..08b8040 100644
--- a/pandas/tests/tseries/test_timezones.py
+++ b/pandas/tests/tseries/test_timezones.py
@@ -1280,6 +1280,25 @@ class TestTimeZones(tm.TestCase):
             self.assertEqual(result_pytz.to_pydatetime().tzname(),
                              result_dateutil.to_pydatetime().tzname())
 
+    def test_tzreplace_issue_15683(self):
+        """Regression test for issue 15683."""
+        dt = datetime(2016, 3, 27, 1)
+        tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo
+
+        result_dt = dt.replace(tzinfo=tzinfo)
+        result_pd = Timestamp(dt).replace(tzinfo=tzinfo)
+
+        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
+        self.assertEqual(result_dt, result_pd)
+        self.assertEqual(result_dt, result_pd.to_pydatetime())
+
+        result_dt = dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
+        result_pd = Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
+
+        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
+        self.assertEqual(result_dt, result_pd)
+        self.assertEqual(result_dt, result_pd.to_pydatetime())
+
     def test_index_equals_with_tz(self):
         left = date_range('1/1/2011', periods=100, freq='H', tz='utc')
         right = date_range('1/1/2011', periods=100, freq='H', tz='US/Eastern')
@jreback

This comment has been minimized.

Copy link
Contributor

commented Apr 24, 2017

ok if u want to put a PR

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Jul 5, 2017

Fix issue pandas-dev#15683
Adjustments to coding guidelines

Only perform timestamp() check if meth is available.

Added sv

jreback added a commit to jreback/pandas that referenced this issue Sep 13, 2017

jreback added a commit to jreback/pandas that referenced this issue Sep 18, 2017

jreback added a commit to jreback/pandas that referenced this issue Sep 20, 2017

jreback added a commit that referenced this issue Sep 20, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.