Please reconsider 'guessing' whether a timestamp is in milliseconds or seconds

[ariebovenberg]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

The following surprising behavior recently tripped me up. When parsing datetimes from unix timestamps, Pydantic
determines automagically whether you mean seconds or milliseconds:

(from the docs)

int or float; assumed as Unix time, i.e. seconds (if >= -2e10 and <= 2e10) or
milliseconds (if < -2e10or > 2e10) since 1 January 1970

This seems sensible at first...but if you're working with milliseconds, Pydantic will decide that most timestamps in 1970 and 1969 are just too small, and they will be interpreted as seconds instead — leading to wildly different times.

While most Pydantic users won't encounter this, it is not a stretch to imagine some Pydantic users are handling data from the 1970s and their validation library, ironically, is silently changing their timestamps.

If you'd rather not change this, please consider adding a big red warning to the docs on the implications of this behavior on timestamps in 1970 and 1969.

Example Code

from datetime import datetime
from pydantic import BaseModel

class Foo(BaseModel):
    a: datetime

# some timestamps in ms
recent = datetime(2023, 10, 4, 12, 22).timestamp() * 1000
apollo17_launch = datetime(1972, 12, 7, 5, 33).timestamp() * 1000
apollo13_launch = datetime(1970, 4, 11, 19, 13).timestamp() * 1000

print(Foo(a=recent))           # 2023-10-04 (correct)
print(Foo(a=apollo17_launch))  # 1972-12-07 (correct)
print(Foo(a=apollo13_launch))  # 2245-11-14 (INCORRECT)

Python, Pydantic & OS Version

pydantic version: 2.4.2
        pydantic-core version: 2.10.1
          pydantic-core build: profile=release pgo=false
                 install path: /Users/arie/.pyenv/versions/3.12.0/envs/sandbox312/lib/python3.12/site-packages/pydantic
               python version: 3.12.0 (main, Oct  3 2023, 08:33:34) [Clang 15.0.0 (clang-1500.0.40.1)]
                     platform: macOS-14.0-arm64-arm-64bit
             related packages: typing_extensions-4.8.0

[samuelcolvin]

This has been around since the very earliest days of pydantic, it has saved me personally hours in combined time not having to thing about whether something is in seconds or milliseconds.

I won't be changing the default.

But I would consider a config switch or similar to stop using the watershed.

[ariebovenberg]

Thanks for the quick reply.

I will certainly agree the behavior is more convenient 😄, and probably saves many developers a lot of time.

Although I am curious: is there another validation library or language that also does it this way? Part of the reason for my bemusement is that I've never encountered this type of behavior before 😅.

Regarding a potential config switch: how about an option called timestamp_unit which can be set to either "auto" (the current behavior and ongoing default), "s", or "ms".

[samuelcolvin]

Solution sounds good to me, "infer" or "guess" might be better than "auto".

[samuelcolvin]

I'm just realized you're the he same person who asked the question on HN, sorry for my dumb initial response. 🙏

By the way, to fix this I think you'll need to update speedate, then pydantic-core.

[ariebovenberg]

I'm just realized you're the he same person who asked the question on HN, sorry for my dumb initial response. 🙏

Thanks for mentioning. Combined with your twitter post today regarding issue submission quality, I was concerned I hadn't brought it up correctly.
In any case...the HN post wasn't the lively discussion I had hoped it to be 🦗.

Solution sounds good to me, "infer" or "guess" might be better than "auto".

"infer" has a nice, neutral sound to it.

I won't be changing the default.

I do feel compelled to say that I disagree with keeping the default behavior like this. If you've already made up your mind, I won't argue — it's your project after all, and there might be some context I'm missing.
But would you be open to putting this question to the community/contributors in some way?

Part of me really wants to write a blog post to convince the world 🔥 😉 , but it's a lot more constructive to work with you all on this 🤝

By the way, to fix this I think you'll need to update speedate, then pydantic-core.

Happy to submit the PRs — if you're OK with a slow pace. I'll need to brush up on Rust, and the codebase in general.

[samuelcolvin]

No you definitely weren't the trigger for my twitter idea.

I'm not pretending this is a democracy, and I'm not resorting to the tyranny of the masses / hegemony of the upvote.

More or the point, we definitely can't change the behaviour anyway until V3.

Obviously happy to accept the config switch asap.

The only thing I'd consider in V3 is to move away from a single watershed, to a ranges which are accepted, with errors outside those ranges. That would mean it doesn't "suddenly" change.

E.g. timestamps are only accepted 1980-2100 or whatever in "infer" mode.