Combined behavior of datetime.datetime.timestamp() and datetime.datetime.utcnow() on non-UTC timezoned machines

[YiLuan]

BPO	39970
Nosy	@abalkin, @pganssle
Superseder	bpo-33293: Using datetime.datetime.utcnow().timestamp() in Python3.6.0 can't get correct UTC timestamp.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-15.16:04:37.438>
created_at = <Date 2020-03-15.15:01:14.193>
labels = ['3.7', 'type-bug', 'library']
title = 'Combined behavior of datetime.datetime.timestamp() and datetime.datetime.utcnow() on non-UTC timezoned machines'
updated_at = <Date 2020-03-17.09:12:56.655>
user = 'https://bugs.python.org/YiLuan'

bugs.python.org fields:

activity = <Date 2020-03-17.09:12:56.655>
actor = 'Yi Luan'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-15.16:04:37.438>
closer = 'p-ganssle'
components = ['Library (Lib)']
creation = <Date 2020-03-15.15:01:14.193>
creator = 'Yi Luan'
dependencies = []
files = []
hgrepos = []
issue_num = 39970
keywords = []
message_count = 8.0
messages = ['364240', '364243', '364244', '364245', '364282', '364283', '364323', '364398']
nosy_count = 3.0
nosy_names = ['belopolsky', 'p-ganssle', 'Yi Luan']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '33293'
type = 'behavior'
url = 'https://bugs.python.org/issue39970'
versions = ['Python 3.7']

[YiLuan]

Hello,

Apologies if this was a duplicate issue.

I guess the most concise way of saying this is that when doing:

>> datetime.datetime.utcnow().timestamp()

on a machine whose local time isn't the UTC time, the above code will not return the correct timestamp.

Because datetime.datetime.timestamp() and datetime.datetime.fromtimestamp() will intrinsically convert the timestamp based on the local time of the running machine, when fed with data that are already converted to UTC, these functions will double convert them hence will return incorrect result.

For example:
On a machine that is in CST time:
>>> dt = datetime.datetime.utcnow()
>>> dt
datetime.datetime(2020, 3, 15, 14, 33, 10, 213664)
>>> datetime.datetime.fromtimestamp(dt.timestamp(), datetime.timezone.utc)
datetime.datetime(2020, 3, 15, 6, 33, 10, 213664)

Meanwhile, on a machine that is in UTC time:
>>> dt = datetime.datetime.utcnow()
>>> dt
datetime.datetime(2020, 3, 15, 14, 41, 2, 203275)
>>> datetime.datetime.fromtimestamp(dt.timestamp(), datetime.timezone.utc)
datetime.datetime(2020, 3, 15, 14, 41, 2, 203275)

I understand that one should probably use datetime.datetime.fromtimestamp() to construct time, but the output of the above code is inconsistent on machines that are set to different timezones. The above code explicitly asked to get the UTC time now and get the timestamp, then convert from a UTC timestamp to a datetime object. The result should be the same on the first machine but it didn't.

From my point of view, timestamp() functions should not shift any datetime objects since it returns an object that is naive about the tzinfo anyway. Timestamp data generated by Python should be correct and code should do what the programmer asked the code to do. In the above example, datetime.datetime.utcnow().timestamp() should return the timestamp of now in UTC time but in fact on a machine in CST time it would return the timestamp 8 hours before the UTC timestamp of now.

The intrinsic behavior of timestamp() functions will cause ambiguity in code, therefore I suggest that timestamp() functions, unless used on tz aware objects, should not shift any date time based on the running machine's local time.

[pganssle]

This is the intended behavior of these functions, and there is actually now a warning on both the utcnow and utcfromtimestamp functionsto reflect this:

https://docs.python.org/3/library/datetime.html#datetime.datetime.utcnow

I would say that the correct answer here is to stop using utcnow and utcfromtimestamp (except possibly in very limited circumstance), I have written about it here:

https://blog.ganssle.io/articles/2019/11/utcnow.html

The preferred way to do this is datetime.now(tzinfo=datetime.timezone.utc) or datetime.fromtimestamp(ts, tzinfo=datetime.timezone.utc).

The main thing to internalize is that the result of .timestamp() always has a time zone, because it is an epoch time, meaning that it is the number of seconds in UTC since 1970-01-01T00:00:00Z.

In Python 2, any operations on naive datetimes that required them to represent absolute times were an error, but in Python 3 that was changed and they were treated as local times. Perhaps leaving that behavior as is and having a dedicated "local time" object would have been a good idea, but there are actually some serious problems with doing it that way because it's difficult to define "local time" in such a way that it may not change over the course of an interpreter lifetime, which would cause major issues for an aware datetime (guaranteed not to change over the course of the interpreter lifetime). Treating naive times as local for operations that require localization (without changing their equality and comparison semantics, which is what would cause the problems) is a neat solution to that.

Sorry this causes confusion, perhaps in the future we can look into removing the .utcnow() and .utcfromtimestamp() functions, or renaming them to something else.

I'm going to set the status of this as wontfix because this is an intended behavior, but feel free to continue to use the ticket for discussion.

Thank you for taking the time to file an issue!

[abalkin]

I am sure this has been reported before – I will try to find the relevant issue. This behavior is correct and documented. The only improvement that we can consider is to make it more explicit that utcnow is deprecated and the correct way to obtain the UTC timestamp is

datetime.now(timezone.utc).timestamp()

[abalkin]

This is a duplicate of bpo-33293.

[YiLuan]

Hi,

Thanks for taking time to reply my question.

I suppose the title I put was a bit confusing.

And the recommended way to generate time in UTC does solve this issue, in Python.

However the message I was trying to convey is, the behavior of timestamp() is one action too much from my point of view.

Say, for example, if I'm sending data generated on my local machine (in CST time) to someone else, and for easy comparison and precision I used datetime.datetime.now().timestamp() as the value of time fields. I would naturally think that the timestamp() function returned the timestamp of my local time. And if the person who received the data, instead of using Python, used Node.js to import those timestamps, and told by me that the timestamps were in CST time. Then he/she will get the wrong time since those timestamps were actually in UTC.
As Scott Mayer, the author of Effective C++, once said, "APIs should be easy to use correctly and hard to use incorrectly", timestamp() functions from my point of view did one thing too much, it shouldn't shift any datetime object that it was fed into.
And in the documentation of datetime.timestamp(), there is no warning about this behavior, only a note on getting the UTC time. It only says "Return POSIX timestamp corresponding to the datetime instance". From my understanding POSIX timestamp is the time elapsed since epoch, not time elapsed -8 hours on CST timed machines since epoch.

By it's nature, timestamps couldn't and shouldn't incorporate any timezone information, when only presented with a naive datetime, program should only convert it to whatever datetime that timestamp represents, rather than thinking that the machine is in some timezone and shifts it by some time and then converts it.

Sorry if what I wrote does not make sense, the reason for me to use utcnow() is just to demonstrate, from my point of view utcnow() or utcfromtimestamp() did nothing wrong, but the problem lies with the timestamp() and fromtimestamp() functions' extra behavior.

Another example is, say, the current unix epoch is 1584330809, when I fed it into Python on my local machine, I get:
>>> datetime.datetime.fromtimestamp(1584330809)
datetime.datetime(2020, 3, 16, 11, 53, 29)

when I fed it into Node.js, I get:

new Date(1584330809*1000)
2020-03-16T03:53:29.000Z

From my point of view, the Node's behavior is much more natural and intuitive.

since we didn't know what timezone 1584330809 is in, the returned datetime should be just how many seconds elapsed since epoch, not my local time of the machine.

[YiLuan]

Sorry to make changes again but I typed his name wrong = =!
It's Scott Meyers. Apologies.

[pganssle]

@yi Luan

I think you may misunderstand what the .timestamp() function does - it returns an epoch time, which is the amount of time (in seconds) elapsed since the Unix epoch: https://en.wikipedia.org/wiki/Unix_time

The number is not different depending on your time zone:

>>> from datetime import *
>>> from dateutil import tz

    >>> dt = datetime(2019, 1, 1, tzinfo=timezone.utc)
    >>> print(f"{dt}: {dt.timestamp()}")
    2019-01-01 00:00:00+00:00: 1546300800.0

    >>> dt = dt.astimezone(tz.gettz("America/New_York"))
    >>> print(f"{dt}: {dt.timestamp()}")
    2018-12-31 19:00:00-05:00: 1546300800.0

    >>> dt = dt.astimezone(tz.gettz("Asia/Tokyo"))
    >>> print(f"{dt}: {dt.timestamp()}")
    2019-01-01 09:00:00+09:00: 1546300800.0

Note how the timestamp number is always the same.

Alexander's suggestion of using datetime.now(tz=timezone.utc).timestamp() is slightly misleading because datetime.now().timestamp() and datetime.now(tz=timezone.utc).timestamp() will always return the same value. I think he was just using that as shorthand for "replace datetime.utcnow() with datetime.now(tz=timezone.utc) in all cases".

When you have a naive datetime (with no tzinfo), the only options are to pick the time zone it represents and convert to UTC or to throw an error and say, "We don't know what time zone this represents, so we cannot do this operation." Python 2 used to throw an exception, but in Python 3 naive datetimes represent local times.

If you want "nominal number of seconds since 1970-01-01T00:00:00 *in this time zone*", you want something more like this:

  def seconds_since(dt, epoch=datetime(1970, 1, 1)):
    return (dt.replace(tzinfo=None) - epoch).total_seconds()

That does not take into account total elapsed time from DST transitions and the like - to do that, you'll want something more like this:

  def seconds_elapsed_since(dt, epoch=datetime(1970, 1, 1)):
    if epoch.tzinfo is None and dt.tzinfo is not None:
        epoch = epoch.replace(tzinfo=dt.tzinfo)
    return (dt - epoch).total_seconds()

I urge you not to do this in any sort of interop protocol, though, because integer timestamps are traditionally interpreted as Unix times, and if you start passing around an integer timestamp that represents "unix time plus or minus a few hours", you are likely to create bugs when someone mistakes it for a unix time.