Incompatibility with datetutil.rrule and pytz.timezone?

[geier]

For some time I've been using pytz's timezone implementation together with dateutil's rrule and have stumbled on an incompatibility several times. When expanding a RRULE over DST boundaries, those DST transition are disregarded (see below for an example). Is this a general incompatibility between those two packages, am I doing it wrong or is there a bug hidden somewhere (either in dateutil or pytz)?

Example

import pytz
from datetime import datetime
from dateutil.rrule import rrulestr

BERLIN = pytz.timezone('Europe/Berlin')
local_time = BERLIN.localize(datetime(2015, 4, 8, 19))
date_list = list(rrulestr('FREQ=MONTHLY;COUNT=12', dtstart=local_time))
# translating these times to utc and back to berlin makes the error obvious
print([dtime.astimezone(pytz.UTC).astimezone(BERLIN) for dtime in date_list])

outputs:

[datetime.datetime(2015, 4, 8, 19, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>), 
datetime.datetime(2015, 5, 8, 19, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>), 
datetime.datetime(2015, 6, 8, 19, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>), 
datetime.datetime(2015, 7, 8, 19, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>), 
datetime.datetime(2015, 8, 8, 19, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>), 
datetime.datetime(2015, 9, 8, 19, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>), 
datetime.datetime(2015, 10, 8, 19, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>), 
datetime.datetime(2015, 11, 8, 18, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CET+1:00:00 STD>), 
datetime.datetime(2015, 12, 8, 18, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CET+1:00:00 STD>), 
datetime.datetime(2016, 1, 8, 18, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CET+1:00:00 STD>), 
datetime.datetime(2016, 2, 8, 18, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CET+1:00:00 STD>), 
datetime.datetime(2016, 3, 8, 18, 0, tzinfo=<DstTzInfo 'Europe/Berlin' CET+1:00:00 STD>)]

doing the same with a dateutil timezone works fine:

from datetime import datetime
from dateutil.rrule import rrulestr
from dateutil import tz

BERLIN = tz.gettz('Europe/Berlin')
local_time = datetime(2015, 4, 8, 19, tzinfo=BERLIN)
date_list = list(rrulestr('FREQ=MONTHLY;COUNT=12', dtstart=local_time))
print([dtime.astimezone(tz.tzutc()).astimezone(BERLIN) for dtime in date_list])

outputs:

[datetime.datetime(2015, 4, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2015, 5, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2015, 6, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2015, 7, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2015, 8, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2015, 9, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2015, 10, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2015, 11, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2015, 12, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2016, 1, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2016, 2, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin')), 
datetime.datetime(2016, 3, 8, 19, 0, tzinfo=tzfile('/usr/share/zoneinfo/Europe/Berlin'))]

[pganssle]

I am pretty sure this is at least partially a duplicate of #88. I think the dateutil timezones work because the utc offsets are more lazily calculated than in pytz, where they are cached after the first invocation. I'll have to dig in a bit more deeply to make sure, but this may be an issue with pytz - or at least just an improper usage of pytz time zones.

I'm guessing that the correct behavior here is that the rule should generate 12 dates with the 8th of each month, at 19:00. One easy solution is to start local_time as a naive datetime with no time zone and apply the timezone afterwards, e.g.:

import pytz
from datetime import datetime
from dateutil import rrule as rr
from dateutil.tz import gettz

try:
    from itertools import imap
except ImportError:
    imap = map

BERLIN = pytz.timezone("Europe/Berlin")

local_time = datetime(2015, 4, 8, 19, 0)
local_time_du = datetime(2015, 4, 8, 19, 0, tzinfo=gettz("Europe/Berlin"))
local_time_pt = BERLIN.localize(datetime(2015, 4, 8, 19, 0))

rrn = rr.rrule(freq=rr.MONTHLY, count=12, dtstart=local_time)       # Naive rule
rrdu = rr.rrule(freq=rr.MONTHLY, count=12, dtstart=local_time_du)   # tzfile
rrpt = rr.rrule(freq=rr.MONTHLY, count=12, dtstart=local_time_pt)   # tzfile
rrm = imap(BERLIN.localize, rrn)   # Map the localizing function to the naive

comp_rrule = lambda rr1, rr2: [cdu == cpt for cdu, cpt in zip(rr1, rr2)]

print("Compare pytz to dateutil:")
print(comp_rrule(rrpt, rrdu))

print("Compare naive->pytz mapping to dateutil:")
print(comp_rrule(rrm, rrdu))

This returns:

Compare pytz to dateutil:
[True, True, True, True, True, True, True, False, False, False, False, False]
Compare naive->pytz mapping to dateutil:
[True, True, True, True, True, True, True, True, True, True, True, True]

I think there are other options as well, such as wrapping pytz in a tzinfo subclass, or subclassing pytz.tzinfo.DstTzInfo.

[stefs]

Reference to an effect of this issue in matplotlib: matplotlib/matplotlib#2737

[stub42]

pytz internally uses fixed offset timezones, swapping in the 'correct' one after arithmetic is completed. This allows it to reliably do timeline arithmetic across DST boundaries using Python's naive arithmetic (because with fixed offset timezones, they are the same thing). Arithmetic is done using the starting point's timezone rules (so DST in the original example). Adding one month will add one month in DST. When this is 'normalized' (in this case, implicitly by astimezone), the timezone is corrected to STD and you lose an hour.

I think the best way of mixing rrule and pytz is to strip the tzinfo using .replace(tzinfo=None), add the rrule, and then timezone.localize(result) to put the timezone back.

import pytz
from datetime import datetime
from dateutil.rrule import rrulestr

BERLIN = pytz.timezone('Europe/Berlin')
naive_local_time = datetime(2015, 4, 8, 19)
date_list = list(rrulestr('FREQ=MONTHLY;COUNT=12', dtstart=naive_local_time))
print([BERLIN.localize(dtime) for dtime in date_list])

[pganssle]

OK, I'm resolving this as wontfix.