datetime in Python 3.6+ no longer respects 'TZ' environment variable

[adamwill]

BPO	30062
Nosy	@abalkin, @jwilk, @ammaraskar, @AdamWill

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-06-25.05:22:50.374>
created_at = <Date 2017-04-12.21:43:42.433>
labels = ['invalid', '3.7', 'library']
title = "datetime in Python 3.6+ no longer respects 'TZ' environment variable"
updated_at = <Date 2017-06-25.05:22:50.356>
user = 'https://github.com/adamwill'

bugs.python.org fields:

activity = <Date 2017-06-25.05:22:50.356>
actor = 'ammar2'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-25.05:22:50.374>
closer = 'ammar2'
components = ['Library (Lib)']
creation = <Date 2017-04-12.21:43:42.433>
creator = 'adamwill'
dependencies = []
files = []
hgrepos = []
issue_num = 30062
keywords = []
message_count = 4.0
messages = ['291574', '291576', '295642', '296803']
nosy_count = 4.0
nosy_names = ['belopolsky', 'jwilk', 'ammar2', 'adamwill']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30062'
versions = ['Python 3.6', 'Python 3.7']

[adamwill]

I can't figure out yet why this is, but it's very easy to demonstrate:

[adamw@adam anaconda (time-log %)]$ python35 
Python 3.5.2 (default, Feb 11 2017, 18:09:24) 
[GCC 7.0.1 20170209 (Red Hat 7.0.1-0.7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import datetime
>>> os.environ['TZ'] = 'America/Winnipeg'
>>> datetime.datetime.fromtimestamp(0)
datetime.datetime(1969, 12, 31, 18, 0)
>>> os.environ['TZ'] = 'Europe/London'
>>> datetime.datetime.fromtimestamp(0)
datetime.datetime(1970, 1, 1, 1, 0)
>>> 

[adamw@adam anaconda (time-log %)]$ python3
Python 3.6.0 (default, Mar 21 2017, 17:30:34) 
[GCC 7.0.1 20170225 (Red Hat 7.0.1-0.10)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import datetime
>>> os.environ['TZ'] = 'America/Winnipeg'
>>> datetime.datetime.fromtimestamp(0)
datetime.datetime(1969, 12, 31, 16, 0)
>>> os.environ['TZ'] = 'Europe/London'
>>> datetime.datetime.fromtimestamp(0)
datetime.datetime(1969, 12, 31, 16, 0)
>>> 

That is, when deciding what timezone to use for operations that involve one, if the 'TZ' environment variable was set, Python 3.5 would use the timezone it was set to. Python 3.6 does not, it ignores it.

As you can see, if I twiddle the 'TZ' setting and call datetime.datetime.fromtimestamp(0) repeatedly under Python 3.5, I get different results - each one is the wall clock time at the epoch (timestamp 0) in the timezone specified as 'TZ'. If I do the same on Python 3.6, the 'TZ' setting is ignored and I always get the same result (the wall clock time of 'the epoch' in Vancouver, which is my real timezone, and which I guess is being picked up from /etc/localtime or whatever).

This wound up causing a problem in the Fedora / Red Hat installer, anaconda:

https://bugzilla.redhat.com/show_bug.cgi?id=1433560

The 'current time zone' can be changed in anaconda. Shortly after it starts up, it automatically tries to guess the correct time zone via geolocation, and the user can also explicitly choose a timezone in the installer interface (or set one in a kickstart). Whenever the timezone is set in this way, an underlying library (libtimezonemap - https://launchpad.net/timezonemap) sets 'TZ' to the chosen timezone. It turns out other code in anaconda relies on Python respecting that setting, which Python 3.6 does not do. As a consequence, anaconda with Python 3.6 winds up setting the system time incorrectly. Also, the timestamps on all its log files are different now, and there may well be other consequences I didn't figure out yet.

The same applies to, e.g., datetime.datetime.now(): you can perform the same experiment with Python 3.5 and 3.6. If you change the 'TZ' env var while calling datetime.datetime.now() after each change, on Python 3.5, the naive datetime object it returns is the current time in that timezone. On Python 3.6, regardless of what 'TZ' is set to, it always gives you the same time.

Is this an intended and/or desired change that we should adjust to somehow? Is there another way a running Python process can change what "the current" timezone is, for the purposes of datetime calculations like this?

[adamwill]

Hmm, after a bit more poking I found this:

https://docs.python.org/3/library/time.html#time.tzset

"Note

Although in many cases, changing the TZ environment variable may affect the output of functions like localtime() without calling tzset(), this behavior should not be relied on."

It seems like that's kinda what we're dealing with here. If I extend my tests to change TZ, call the test function, then call time.tzset() and call the test function again, the second call to the test function gives the different result, i.e. the time.tzset() call does what it claims and changes Python's conception of the 'current' timezone.

So while that note has been there all along, it seems like the behaviour actually changed between 3.5 and 3.6, and a change to 'TZ' is now less likely to be respected without a tzset() call. But given the doc note, perhaps that can't be considered a bug.

anaconda doesn't call time.tzset() anywhere at present. It's also multi-threaded, so making sure all the threads call time.tzset() after any thread has changed what the 'current' timezone is will be lots of fun to implement, I guess :/

[jwilk]

This is a side-effect of fixing issue bpo-28067.

[ammaraskar]

Closing this since as you pointed out this behavior is already documented.

As an aside, seeing as you guys already have pytz as a dependency. Instead of fixing this with tzset you can also just explicitly do datetime.datetime.fromtimestamp(time, pytz.timezone(os.environ['TZ']))