Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure with Cython 3 TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian' #271

Closed
opoplawski opened this issue Feb 27, 2022 · 11 comments
Closed

Test failure with Cython 3 TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian' #271

opoplawski opened this issue Feb 27, 2022 · 11 comments

Comments

@opoplawski
Copy link

See https://bugzilla.redhat.com/show_bug.cgi?id=2058169

Version 1.5.2 on Fedora rawhide with Python 3.11.0a5:

=================================== FAILURES ===================================
________________________ TestDate2index.test_select_nc _________________________

self = <test.test_cftime.TestDate2index testMethod=test_select_nc>

    def test_select_nc(self):
        nutime = self.time_vars['time']
    
        # these are python datetimes ('proleptic_gregorian' calendar).
        dates = [datetime(1950, 1, 2, 6), datetime(
            1950, 1, 3), datetime(1950, 1, 3, 18)]
    
        t = date2index(dates, nutime, select='before')
        assert_equal(t, [1, 2, 2])
    
        t = date2index(dates, nutime, select='after')
        assert_equal(t, [2, 2, 3])
    
        t = date2index(dates, nutime, select='nearest')
        assert_equal(t, [1, 2, 3])
    
        # Test dates outside the support with select
        t = date2index(datetime(1949, 12, 1), nutime, select='nearest')
        assert_equal(t, 0)
    
        t = date2index(datetime(1978, 1, 1), nutime, select='nearest')
        assert_equal(t, 365)
    
        # Test dates outside the support with before
        self.assertRaises(
            ValueError, date2index, datetime(1949, 12, 1), nutime, select='before')
    
        t = date2index(datetime(1978, 1, 1), nutime, select='before')
        assert_equal(t, 365)
    
        # Test dates outside the support with after
        t = date2index(datetime(1949, 12, 1), nutime, select='after')
        assert_equal(t, 0)
    
        self.assertRaises(
            ValueError, date2index, datetime(1978, 1, 1), nutime, select='after')
        # test microsecond and millisecond units
        unix_epoch = "milliseconds since 1970-01-01T00:00:00Z"
        d = datetime(2038, 1, 19, 3, 14, 7)
        millisecs = int(
            date2num(d, unix_epoch, calendar='proleptic_gregorian'))
        assert_equal(millisecs, (2 ** 32 / 2 - 1) * 1000)
        unix_epoch = "microseconds since 1970-01-01T00:00:00Z"
        microsecs = int(date2num(d, unix_epoch))
        assert_equal(microsecs, (2 ** 32 / 2 - 1) * 1000000)
        # test microsecond accuracy in date2num/num2date roundtrip
        # note: microsecond accuracy lost for time intervals greater
        # than about 270 years.
        units = 'microseconds since 1776-07-04 00:00:00-12:00'
        dates = [datetime(1962, 10, 27, 6, 1, 30, 9001),
                 datetime(1993, 11, 21, 12, 5, 25, 999),
                 datetime(1995, 11, 25, 18, 7, 59, 999999)]
        times2 = date2num(dates, units)
        dates2 = num2date(times2, units)
>       datediff = abs(dates-dates2)
E       TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian'

test/test_cftime.py:1107: TypeError
______________________________ DateTime.test_add _______________________________

self = <test.test_cftime.DateTime testMethod=test_add>

    def test_add(self):
        dt = self.date1_365_day
        # datetime + timedelta
        self.assertEqual(dt + self.delta, # add 25 hours
                         dt.replace(day=dt.day + 1, hour=dt.hour + 1))
    
        # timedelta + datetime
>       self.assertEqual(self.delta + dt, # add 25 hours
                         dt.replace(day=dt.day + 1, hour=dt.hour + 1))
E       TypeError: unsupported operand type(s) for +: 'datetime.timedelta' and 'cftime._cftime.DatetimeNoLeap'

test/test_cftime.py:1199: TypeError
______________________________ DateTime.test_sub _______________________________

self = <test.test_cftime.DateTime testMethod=test_sub>

    def test_sub(self):
        # subtracting a timedelta
        previous_day = self.date1_365_day - self.delta
        self.assertEqual(previous_day.day, self.date1_365_day.day - 1)
    
        def total_seconds(td):
            """Equivalent to td.total_seconds() on Python >= 2.7. See
            https://docs.python.org/2/library/datetime.html#datetime.timedelta.total_seconds
            """
            return (td.microseconds + (td.seconds + td.days * 24 * 3600) * 10**6) / 10**6
    
        # sutracting two cftime.datetime instances
        delta = self.date2_365_day - self.date1_365_day
        # date1 and date2 are exactly one day apart
        self.assertEqual(total_seconds(delta), 86400)
    
        # subtracting cftime.datetime from datetime.datetime
>       delta = self.datetime_date1 - self.date3_gregorian
E       TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian'

test/test_cftime.py:1254: TypeError
=============================== warnings summary ===============================
../../../../usr/lib64/python3.11/site-packages/numpy/distutils/ccompiler.py:8
  /usr/lib64/python3.11/site-packages/numpy/distutils/ccompiler.py:8: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils import ccompiler

../../../../usr/lib64/python3.11/site-packages/numpy/distutils/ccompiler.py:17
  /usr/lib64/python3.11/site-packages/numpy/distutils/ccompiler.py:17: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
    from distutils.sysconfig import customize_compiler

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED test/test_cftime.py::TestDate2index::test_select_nc - TypeError: unsup...
FAILED test/test_cftime.py::DateTime::test_add - TypeError: unsupported opera...
FAILED test/test_cftime.py::DateTime::test_sub - TypeError: unsupported opera...
================= 3 failed, 2254 passed, 2 warnings in 48.70s ==================

https://docs.python.org/3.11/whatsnew/3.11.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.11/fedora-rawhide-x86_64/03525898-python-cftime/

For all our attempts to build python-cftime with Python 3.11, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/package/python-cftime/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.11:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/

Let us know here if you have any questions.

Python 3.11 is planned to be included in Fedora 37. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.11.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.
@jswhit2
Copy link

jswhit2 commented Mar 2, 2022 •
edited

I've added python 3.11-devel to the github actions for ubuntu-latest, and all the tests pass (#274).

@opoplawski
Copy link
Author

Interesting. Maybe it's fixed in current master? Or there is some other difference between the pythons. Thanks for checking.

@jswhit
Copy link
Collaborator

jswhit commented Mar 3, 2022

It built 3.11.a5. Don't see any changes between 1.5.2 and 1.6.0 that would have affected how __add__ and __sub__ behave.

@jswhit
Copy link
Collaborator

jswhit commented Mar 3, 2022

just made a 1.6.0 release, so you could try that...

@opoplawski
Copy link
Author

Using the Fedora test copr repo I still get the failure with 1.6.0:

________________________ TestDate2index.test_select_nc _________________________

self = <test.test_cftime.TestDate2index testMethod=test_select_nc>

    def test_select_nc(self):
        nutime = self.time_vars['time']
    
        # these are python datetimes ('proleptic_gregorian' calendar).
        dates = [datetime(1950, 1, 2, 6), datetime(
            1950, 1, 3), datetime(1950, 1, 3, 18)]
    
        t = date2index(dates, nutime, select='before')
        assert_equal(t, [1, 2, 2])
    
        t = date2index(dates, nutime, select='after')
        assert_equal(t, [2, 2, 3])
    
        t = date2index(dates, nutime, select='nearest')
        assert_equal(t, [1, 2, 3])
    
        # Test dates outside the support with select
        t = date2index(datetime(1949, 12, 1), nutime, select='nearest')
        assert_equal(t, 0)
    
        t = date2index(datetime(1978, 1, 1), nutime, select='nearest')
        assert_equal(t, 365)
    
        # Test dates outside the support with before
        self.assertRaises(
            ValueError, date2index, datetime(1949, 12, 1), nutime, select='before')
    
        t = date2index(datetime(1978, 1, 1), nutime, select='before')
        assert_equal(t, 365)
    
        # Test dates outside the support with after
        t = date2index(datetime(1949, 12, 1), nutime, select='after')
        assert_equal(t, 0)
    
        self.assertRaises(
            ValueError, date2index, datetime(1978, 1, 1), nutime, select='after')
        # test microsecond and millisecond units
        unix_epoch = "milliseconds since 1970-01-01T00:00:00Z"
        d = datetime(2038, 1, 19, 3, 14, 7)
        millisecs = int(
            date2num(d, unix_epoch, calendar='proleptic_gregorian'))
        assert_equal(millisecs, (2 ** 32 / 2 - 1) * 1000)
        unix_epoch = "microseconds since 1970-01-01T00:00:00Z"
        microsecs = int(date2num(d, unix_epoch))
        assert_equal(microsecs, (2 ** 32 / 2 - 1) * 1000000)
        # test microsecond accuracy in date2num/num2date roundtrip
        # note: microsecond accuracy lost for time intervals greater
        # than about 270 years.
        units = 'microseconds since 1776-07-04 00:00:00-12:00'
        dates = [datetime(1962, 10, 27, 6, 1, 30, 9001),
                 datetime(1993, 11, 21, 12, 5, 25, 999),
                 datetime(1995, 11, 25, 18, 7, 59, 999999)]
        times2 = date2num(dates, units)
        dates2 = num2date(times2, units)
>       datediff = abs(dates-dates2)
E       TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian'

test/test_cftime.py:1111: TypeError
______________________________ DateTime.test_add _______________________________

self = <test.test_cftime.DateTime testMethod=test_add>

    def test_add(self):
        dt = self.date1_365_day
        # datetime + timedelta
        self.assertEqual(dt + self.delta, # add 25 hours
                         dt.replace(day=dt.day + 1, hour=dt.hour + 1))
    
        # timedelta + datetime
>       self.assertEqual(self.delta + dt, # add 25 hours
                         dt.replace(day=dt.day + 1, hour=dt.hour + 1))
E       TypeError: unsupported operand type(s) for +: 'datetime.timedelta' and 'cftime._cftime.DatetimeNoLeap'

test/test_cftime.py:1215: TypeError
______________________________ DateTime.test_sub _______________________________

self = <test.test_cftime.DateTime testMethod=test_sub>

    def test_sub(self):
        # subtracting a timedelta
        previous_day = self.date1_365_day - self.delta
        self.assertEqual(previous_day.day, self.date1_365_day.day - 1)
    
        def total_seconds(td):
            """Equivalent to td.total_seconds() on Python >= 2.7. See
            https://docs.python.org/2/library/datetime.html#datetime.timedelta.total_seconds
            """
            return (td.microseconds + (td.seconds + td.days * 24 * 3600) * 10**6) / 10**6
    
        # sutracting two cftime.datetime instances
        delta = self.date2_365_day - self.date1_365_day
        # date1 and date2 are exactly one day apart
        self.assertEqual(total_seconds(delta), 86400)
    
        # subtracting cftime.datetime from datetime.datetime
>       delta = self.datetime_date1 - self.date3_gregorian
E       TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian'

test/test_cftime.py:1270: TypeError

@opoplawski
Copy link
Author

These went away for a while, but no appear to have returned with the transition to Cython 3:

=================================== FAILURES ===================================
________________________ TestDate2index.test_select_nc _________________________

self = <test.test_cftime.TestDate2index testMethod=test_select_nc>

    def test_select_nc(self):
        nutime = self.time_vars['time']
    
        # these are python datetimes ('proleptic_gregorian' calendar).
        dates = [datetime(1950, 1, 2, 6), datetime(
            1950, 1, 3), datetime(1950, 1, 3, 18)]
    
        t = date2index(dates, nutime, select='before')
        assert_equal(t, [1, 2, 2])
    
        t = date2index(dates, nutime, select='after')
        assert_equal(t, [2, 2, 3])
    
        t = date2index(dates, nutime, select='nearest')
        assert_equal(t, [1, 2, 3])
    
        # Test dates outside the support with select
        t = date2index(datetime(1949, 12, 1), nutime, select='nearest')
        assert_equal(t, 0)
    
        t = date2index(datetime(1978, 1, 1), nutime, select='nearest')
        assert_equal(t, 365)
    
        # Test dates outside the support with before
        self.assertRaises(
            ValueError, date2index, datetime(1949, 12, 1), nutime, select='before')
    
        t = date2index(datetime(1978, 1, 1), nutime, select='before')
        assert_equal(t, 365)
    
        # Test dates outside the support with after
        t = date2index(datetime(1949, 12, 1), nutime, select='after')
        assert_equal(t, 0)
    
        self.assertRaises(
            ValueError, date2index, datetime(1978, 1, 1), nutime, select='after')
        # test microsecond and millisecond units
        unix_epoch = "milliseconds since 1970-01-01T00:00:00Z"
        d = datetime(2038, 1, 19, 3, 14, 7)
        millisecs = int(
            date2num(d, unix_epoch, calendar='proleptic_gregorian'))
        assert_equal(millisecs, (2 ** 32 / 2 - 1) * 1000)
        unix_epoch = "microseconds since 1970-01-01T00:00:00Z"
        microsecs = int(date2num(d, unix_epoch))
        assert_equal(microsecs, (2 ** 32 / 2 - 1) * 1000000)
        # test microsecond accuracy in date2num/num2date roundtrip
        # note: microsecond accuracy lost for time intervals greater
        # than about 270 years.
        units = 'microseconds since 1776-07-04 00:00:00-12:00'
        dates = [datetime(1962, 10, 27, 6, 1, 30, 9001),
                 datetime(1993, 11, 21, 12, 5, 25, 999),
                 datetime(1995, 11, 25, 18, 7, 59, 999999)]
        times2 = date2num(dates, units)
        dates2 = num2date(times2, units)
>       datediff = abs(dates-dates2)
E       TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian'

test/test_cftime.py:1114: TypeError
______________________________ DateTime.test_add _______________________________

self = <test.test_cftime.DateTime testMethod=test_add>

    def test_add(self):
        dt = self.date1_365_day
        # datetime + timedelta
        self.assertEqual(dt + self.delta, # add 25 hours
                         dt.replace(day=dt.day + 1, hour=dt.hour + 1))
    
        # timedelta + datetime
>       self.assertEqual(self.delta + dt, # add 25 hours
                         dt.replace(day=dt.day + 1, hour=dt.hour + 1))
E       TypeError: unsupported operand type(s) for +: 'datetime.timedelta' and 'cftime._cftime.DatetimeNoLeap'

test/test_cftime.py:1218: TypeError
______________________________ DateTime.test_sub _______________________________

self = <test.test_cftime.DateTime testMethod=test_sub>

    def test_sub(self):
        # subtracting a timedelta
        previous_day = self.date1_365_day - self.delta
        self.assertEqual(previous_day.day, self.date1_365_day.day - 1)
    
        def total_seconds(td):
            """Equivalent to td.total_seconds() on Python >= 2.7. See
            https://docs.python.org/2/library/datetime.html#datetime.timedelta.total_seconds
            """
            return (td.microseconds + (td.seconds + td.days * 24 * 3600) * 10**6) / 10**6
    
        # sutracting two cftime.datetime instances
        delta = self.date2_365_day - self.date1_365_day
        # date1 and date2 are exactly one day apart
        self.assertEqual(total_seconds(delta), 86400)
    
        # subtracting cftime.datetime from datetime.datetime
>       delta = self.datetime_date1 - self.date3_gregorian
E       TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian'

test/test_cftime.py:1273: TypeError

@opoplawski opoplawski changed the title Test failure with python 3.11 TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian' Test failure with Cython 3 TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'cftime._cftime.DatetimeGregorian' Jul 20, 2023
@mgorny
Copy link

mgorny commented Jul 28, 2023

Please disregard my last comment. I've forgotten that pip does isolated builds, so it installed Cython-3 for the build x_x.

@spencerkclark
Copy link
Collaborator

Indeed I think we are also hitting errors related to this now in our upstream build in xarray (pydata/xarray#7977), likely because Cython 3.0.0 was officially released a couple weeks ago (previously it was in beta). Specifically we are seeing variants of this one (which appear in the test failures above):

>>> import cftime; import datetime
>>> datetime.timedelta(days=1) + cftime.DatetimeNoLeap(2000, 1, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'datetime.timedelta' and 'cftime._cftime.DatetimeNoLeap'

This section of the migration guide for Cython 3 related to "Arithmetic special methods" seems potentially relevant. If I pin Cython to something less than version 3 in the build dependencies section of the pyproject.toml file, the errors go away.

This was referenced Jul 29, 2023
Closed
Merged
@spencerkclark
Copy link
Collaborator

#305 proposes a possible fix.

@jswhit
Copy link
Collaborator

jswhit commented Jul 29, 2023

Closed by PR #305

@jswhit jswhit closed this as completed Jul 29, 2023
@mgorny
Copy link

mgorny commented Jul 31, 2023

Thank you! I can confirm that this solves the problem for us.

gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue Jul 31, 2023
@mgorny
dev-python/cftime: Backport cython-3 fix
e1505ba 
Bug: Unidata/cftime#271
Closes: https://bugs.gentoo.org/898668
Signed-off-by: Michał Górny <mgorny@gentoo.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mgorny @jswhit @opoplawski @spencerkclark @jswhit2