COMPAT: dateutil fixups for 2.3 (GH9021, GH8639) #9036

jreback · 2014-12-07T20:51:49Z

fixes #9021
but going to reopen #8639

jreback · 2014-12-07T21:18:35Z

ok, this shows the failures (scroll way to the end): https://travis-ci.org/jreback/pandas/jobs/43281972

jreback · 2014-12-07T21:22:54Z

cc @dbew

jreback · 2014-12-07T22:26:10Z

this mostly fixes the issues so going to merge this.

COMPAT: dateutil fixups for 2.3 (GH9021, GH8639)

jlec · 2014-12-08T08:31:10Z

I get massive test failures similar to travis. Any ETA for a fix of those?

jreback · 2014-12-08T10:12:06Z

are u sure that u updated to master and recompiled
their should only be a single test failure ATM

jlec · 2014-12-08T10:38:16Z

Fetching https://github.com/pydata/pandas.git ...
Checking out https://github.com/pydata/pandas.git to /var/tmp/portage/dev-python/pandas-9999/work/pandas-9999 ...
git checkout --quiet master
GIT update -->
repository: https://github.com/pydata/pandas.git
at the commit: 67ec0a8

......

ERROR: test_timestamp_tz_convert (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 341, in test_timestamp_tz_convert
conv = idx[0].tz_convert(self.tzstr('US/Pacific'))
File "pandas/tslib.pyx", line 481, in pandas.tslib.Timestamp.tz_convert (pandas/tslib.c:10835)
TypeError: Cannot convert tz-naive Timestamp, use tz_localize to localize

ERROR: test_utc_with_system_utc (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 849, in test_utc_with_system_utc
self.assertEqual(ts, ts.tz_convert(dateutil.tz.tzutc()))
File "pandas/tslib.pyx", line 481, in pandas.tslib.Timestamp.tz_convert (pandas/tslib.c:10835)
TypeError: Cannot convert tz-naive Timestamp, use tz_localize to localize

ERROR: test_with_tz (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 382, in test_with_tz
central = dr.tz_convert(tz)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/index.py", line 1609, in tz_convert
raise TypeError('Cannot convert tz-naive timestamps, use '
TypeError: Cannot convert tz-naive timestamps, use tz_localize to localize

ERROR: test_tz_convert_roundtrip (test_timezones.TestTimeZones)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 1035, in test_tz_convert_roundtrip
reset = converted.tz_convert(None)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/index.py", line 1609, in tz_convert
raise TypeError('Cannot convert tz-naive timestamps, use '
TypeError: Cannot convert tz-naive timestamps, use tz_localize to localize

FAIL: test_ambiguous_flags (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 471, in test_ambiguous_flags
self.assert_numpy_array_equal(dr, localized)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 638, in assert_numpy_array_equal
'{0} is not equal to {1}.'.format(np_array, assert_equal))
AssertionError: <class 'pandas.tseries.index.DatetimeIndex'>
[2011-11-06 00:00:00, ..., 2011-11-06 04:00:00]
Length: 5, Freq: H, Timezone: None is not equal to <class 'pandas.tseries.index.DatetimeIndex'>
[2011-11-06 00:00:00, ..., 2011-11-06 03:00:00]
Length: 5, Freq: None, Timezone: None.

FAIL: test_ambiguous_infer (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 432, in test_ambiguous_infer
self.assertRaises(pytz.AmbiguousTimeError, dr.tz_localize, tz)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 1562, in assertRaises
_callable(_args, *_kwargs)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 1626, in exit
raise AssertionError("{0} not raised.".format(name))
AssertionError: AmbiguousTimeError not raised.

FAIL: test_ambiguous_nat (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 517, in test_ambiguous_nat
self.assert_numpy_array_equal(di_test, localized)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 638, in assert_numpy_array_equal
'{0} is not equal to {1}.'.format(np_array, assert_equal))
AssertionError: <class 'pandas.tseries.index.DatetimeIndex'>
[2011-11-06 00:00:00-04:00, ..., 2011-11-06 03:00:00-05:00]
Length: 5, Freq: None, Timezone: US/Eastern is not equal to <class 'pandas.tseries.index.DatetimeIndex'>
[2011-11-06 00:00:00, ..., 2011-11-06 03:00:00]
Length: 5, Freq: None, Timezone: None.

FAIL: test_datetimeindex_tz_nat (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 812, in test_datetimeindex_tz_nat
self.assertTrue(idx[0].tzinfo is not None)
AssertionError: False is not true

FAIL: test_frame_no_datetime64_dtype (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 673, in test_frame_no_datetime64_dtype
self.assertEqual(e['B'].dtype, 'O')
AssertionError: dtype('<M8[ns]') != 'O'

FAIL: test_index_with_timezone_repr (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 562, in test_index_with_timezone_repr
self.assertIn('2010-04-13 00:00:00', rng_repr)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 572, in assertIn
assert a in b, "%s: %r is not in %r" % (msg.format(a, b), a, b)
AssertionError: : '2010-04-13 00:00:00' is not in "<class 'pandas.tseries.index.DatetimeIndex'>\n[2010-04-13, ..., 2010-05-06]\nLength: 24, Freq: D, Timezone: None"

FAIL: test_infer_tz (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 538, in test_infer_tz
self.assertRaises(Exception, tools._infer_tzinfo, start, end)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 1562, in assertRaises
_callable(_args, *_kwargs)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 1626, in exit
raise AssertionError("{0} not raised.".format(name))
AssertionError: Exception not raised.

FAIL: test_localize_utc_conversion (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 107, in test_localize_utc_conversion
self.assert_numpy_array_equal(converted.asi8, expected_naive.asi8)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 638, in assert_numpy_array_equal
'{0} is not equal to {1}.'.format(np_array, assert_equal))
AssertionError: [1331337600000000000 1331339400000000000 1331341200000000000
1331343000000000000 1331344800000000000 1331346600000000000
1331348400000000000 1331350200000000000 1331352000000000000
1331353800000000000 1331355600000000000 1331357400000000000
1331359200000000000 1331361000000000000 1331362800000000000
1331364600000000000 1331366400000000000 1331368200000000000
1331370000000000000 1331371800000000000 1331373600000000000
1331375400000000000 1331377200000000000 1331379000000000000
1331380800000000000 1331382600000000000 1331384400000000000
1331386200000000000 1331388000000000000 1331389800000000000
1331391600000000000 1331393400000000000 1331395200000000000
1331397000000000000 1331398800000000000 1331400600000000000
1331402400000000000 1331404200000000000 1331406000000000000
1331407800000000000 1331409600000000000 1331411400000000000
1331413200000000000 1331415000000000000 1331416800000000000
1331418600000000000 1331420400000000000 1331422200000000000
1331424000000000000] is not equal to [1331355600000000000 1331357400000000000 1331359200000000000
1331361000000000000 1331362800000000000 1331364600000000000
1331366400000000000 1331368200000000000 1331370000000000000
1331371800000000000 1331373600000000000 1331375400000000000
1331377200000000000 1331379000000000000 1331380800000000000
1331382600000000000 1331384400000000000 1331386200000000000
1331388000000000000 1331389800000000000 1331391600000000000
1331393400000000000 1331395200000000000 1331397000000000000
1331398800000000000 1331400600000000000 1331402400000000000
1331404200000000000 1331406000000000000 1331407800000000000
1331409600000000000 1331411400000000000 1331413200000000000
1331415000000000000 1331416800000000000 1331418600000000000
1331420400000000000 1331422200000000000 1331424000000000000
1331425800000000000 1331427600000000000 1331429400000000000
1331431200000000000 1331433000000000000 1331434800000000000
1331436600000000000 1331438400000000000 1331440200000000000
1331442000000000000].

FAIL: test_localize_utc_conversion_explicit (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 122, in test_localize_utc_conversion_explicit
self.assert_(np.array_equal(converted.asi8, expected_naive.asi8))
AssertionError: False is not true

FAIL: test_timedelta_push_over_dst_boundary (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 184, in test_timedelta_push_over_dst_boundary
self.assertEquals(result, expected)
AssertionError: Timestamp('2012-03-11 04:00:00') != Timestamp('2012-03-11 05:00:00')

FAIL: test_timedelta_push_over_dst_boundary_explicit (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 197, in test_timedelta_push_over_dst_boundary_explicit
self.assertEqual(result, expected)
AssertionError: Timestamp('2012-03-11 04:00:00') != Timestamp('2012-03-11 05:00:00')

FAIL: test_tz_localize_dti (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 210, in test_tz_localize_dti
self.assert_numpy_array_equal(dti2.values, dti_utc.values)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 638, in assert_numpy_array_equal
'{0} is not equal to {1}.'.format(np_array, assert_equal))
AssertionError: ['2005-01-01T01:00:00.000000000+0100' '2005-01-01T01:00:00.001000000+0100'
'2005-01-01T01:00:00.002000000+0100' ...,
'2005-01-01T01:00:30.254000000+0100' '2005-01-01T01:00:30.255000000+0100'
'2005-01-01T01:00:30.256000000+0100'] is not equal to ['2005-01-01T06:00:00.000000000+0100' '2005-01-01T06:00:00.001000000+0100'
'2005-01-01T06:00:00.002000000+0100' ...,
'2005-01-01T06:00:30.254000000+0100' '2005-01-01T06:00:30.255000000+0100'
'2005-01-01T06:00:30.256000000+0100'].

FAIL: test_utc_box_timestamp_and_localize (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 335, in test_utc_box_timestamp_and_localize
self.assert_('EDT' in repr(rng_eastern[0].tzinfo) or 'tzfile' in repr(rng_eastern[0].tzinfo))
AssertionError: False is not true

FAIL: test_with_tz_ambiguous_times (test_timezones.TestTimeZoneSupportDateutil)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 411, in test_with_tz_ambiguous_times
self.assertRaises(pytz.NonExistentTimeError, dr.tz_localize, tz)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 1562, in assertRaises
_callable(_args, *_kwargs)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 1626, in exit
raise AssertionError("{0} not raised.".format(name))
AssertionError: NonExistentTimeError not raised.

FAIL: test_tz_localize_roundtrip (test_timezones.TestTimeZones)

Traceback (most recent call last):
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/tseries/tests/test_timezones.py", line 964, in test_tz_localize_roundtrip
localized.tz_localize(tz)
File "/var/tmp/portage/dev-python/pandas-9999/work/pandas-9999-python2_7/lib/pandas/util/testing.py", line 1626, in exit
raise AssertionError("{0} not raised.".format(name))
AssertionError: TypeError not raised.

Ran 135 tests in 0.524s

FAILED (SKIP=2, errors=4, failures=15)

jlec · 2014-12-08T10:39:47Z

INSTALLED VERSIONS

commit: None
python: 2.7.8.final.0
python-bits: 64
OS: Linux
OS-release: 3.17.4-lh
machine: x86_64
processor: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
byteorder: little
LC_ALL: None
LANG: en_GB.utf8

pandas: 0.15.1-213-g67ec0a8
nose: 1.3.4
Cython: 0.21.1
numpy: 1.8.2
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 2.3.1
sphinx: 1.2.3
patsy: 0.3.0
dateutil: 2.3
pytz: 2014.10
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.4
matplotlib: 1.4.2
openpyxl: 2.1.0
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: None
lxml: 3.4.1
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: None
pymysql: None
psycopg2: None

jreback · 2014-12-08T10:51:30Z

you need to

python setup.py build_ext --inplace

you didn't recompile

I was able to repro the problem, and the single original test failure remains (but is actually a larger problem).

datetutil 2.3 broke the API (again :()), this was release a few days ago

easiest is simply to downgrade to 2.2. or use pytz timezones which are the default anyhow

jlec · 2014-12-08T11:00:57Z

I did recompile, but I didn't paste the log here.
In case you would like to see it, it is here http://dev.gentoo.org/~jlec/paste/build.log

jreback · 2014-12-08T11:13:11Z

you are showing me a build log for 3.3.
not really sure what that is anyhow.

Pls humor me. Just clone the repo in a new directly, compile and test on 2.7.

Where are you seeing a travis failure (again except for the final build, which has 2 test failures)?

jlec · 2014-12-08T11:35:05Z

Just scroll up, there is the test failure for 2.7.

And believe me, I am experienced enough in building packages that you don't need to treat me as a stupid noob. The failure is there on python 2.7, 3.3 and 3.4.

jreback · 2014-12-08T11:44:48Z

@jlec I appreciate you are trying to help. Pls downgrade to 2.2
and await fix when someone gets around to it. Complain to dateutil that they broke the API.

jlec · 2014-12-08T13:21:06Z

This is plain python-2.7 with current HEAD and python-dateutils-2.2

https://gist.github.com/jlec/5dc56c919e5a0506dcbb

jreback · 2014-12-08T13:40:56Z

so since this is not happening on any other linux (for 2.2), then something different between what you are doing and say travis. maybe you can help figure that out.

IIRC you had this problem with 0.15.0 as well?

jlec · 2014-12-08T14:14:42Z

With 0.15* I had #8639

But I am retesting right now.

jlec · 2014-12-08T14:46:53Z

So current HEAD and 0.15.1 show different problems using dateutil-2.2

jreback added Compat pandas objects compatability with Numpy or Python functions Testing pandas testing functions or related to the test suite labels Dec 7, 2014

jreback added this to the 0.15.2 milestone Dec 7, 2014

This was referenced Dec 7, 2014

pandas-0.15.0 test failure test_string_index_alias_tz_aware (pandas.tseries.tests.test_timezones.TestTimeZoneSupportDateutil) #8639

Closed

TST: dateutil 2.3 #9021

Closed

COMPAT: dateutil fixups for 2.3 (GH9021, GH8639)

4e4640f

jreback force-pushed the dateutil_fix branch from 24daba1 to 4e4640f Compare December 7, 2014 21:22

jreback added a commit that referenced this pull request Dec 7, 2014

Merge pull request #9036 from jreback/dateutil_fix

73ca1f9

COMPAT: dateutil fixups for 2.3 (GH9021, GH8639)

jreback merged commit 73ca1f9 into pandas-dev:master Dec 7, 2014

jarondl mentioned this pull request Dec 10, 2014

pandas claim we broke the API dateutil/dateutil#5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

COMPAT: dateutil fixups for 2.3 (GH9021, GH8639) #9036

COMPAT: dateutil fixups for 2.3 (GH9021, GH8639) #9036

jreback commented Dec 7, 2014

jreback commented Dec 7, 2014

jreback commented Dec 7, 2014

jreback commented Dec 7, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jlec commented Dec 8, 2014

COMPAT: dateutil fixups for 2.3 (GH9021, GH8639) #9036

COMPAT: dateutil fixups for 2.3 (GH9021, GH8639) #9036

Conversation

jreback commented Dec 7, 2014

jreback commented Dec 7, 2014

jreback commented Dec 7, 2014

jreback commented Dec 7, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

ERROR: test_timestamp_tz_convert (test_timezones.TestTimeZoneSupportDateutil)

ERROR: test_utc_with_system_utc (test_timezones.TestTimeZoneSupportDateutil)

ERROR: test_with_tz (test_timezones.TestTimeZoneSupportDateutil)

ERROR: test_tz_convert_roundtrip (test_timezones.TestTimeZones)

FAIL: test_ambiguous_flags (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_ambiguous_infer (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_ambiguous_nat (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_datetimeindex_tz_nat (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_frame_no_datetime64_dtype (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_index_with_timezone_repr (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_infer_tz (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_localize_utc_conversion (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_localize_utc_conversion_explicit (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_timedelta_push_over_dst_boundary (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_timedelta_push_over_dst_boundary_explicit (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_tz_localize_dti (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_utc_box_timestamp_and_localize (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_with_tz_ambiguous_times (test_timezones.TestTimeZoneSupportDateutil)

FAIL: test_tz_localize_roundtrip (test_timezones.TestTimeZones)

jlec commented Dec 8, 2014

INSTALLED VERSIONS

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jreback commented Dec 8, 2014

jlec commented Dec 8, 2014

jlec commented Dec 8, 2014