`OverflowError` using `datetime.fromtimestamp` with a large negative number

[samuelcolvin]

🐛 Bug

OverflowError using datetime.fromtimestamp with a large negative number

To Reproduce

Using https://pyodide.org/en/stable/console.html

Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr  9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import datetime
>>> datetime.fromtimestamp(-2147483649)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
OverflowError: timestamp out of range for platform time_t

I also get the error using node, see this build

Expected behavior

Outside wasm:

In [1]: datetime.fromtimestamp(-2147483649)
Out[1]: datetime.datetime(1901, 12, 13, 20, 9, 6)

Environment

• Pyodide Version: 0.21.0-alpha.2, also on https://pyodide.org/en/stable/console.html (Is there a way to get pyiodide version inside the interpreter? I tried sys.version and platform, but couldn't fine it
• Browser version: Version 103.0.5060.53 (Official Build) (arm64) and also node v18 on x86
• Any other relevant information: See this CI run

[hoodmane]

Well Python's datetime type uses the system's time_t size which is defined by Emscripten to be 32 bit. @sbc100 is there a good reason time_t has to be 32 bits rather than 64 bits? The 32 bit time range from 1902 to 2038 is a bit narrow.

[sbc100]

I think the choice is due to the fact the i32 is much easier to deal with at the wasm/JS boundary. At least historically. We now have BigInt integration but it is not yet on by default, so where it we can we try to prefer to avoid i64 at the syscall layer.

Can python not handle time_t being defined as 32-bit? Seems like it should not be making assumptions about that width of time_t.

[sbc100]

Having said that I think we recently refactored the syscall layer such that time_t is not longer part of the API...so we could make this change...

[hoodmane]

Python itself doesn't make assumptions about the size of time_t but it exposes it directly to users and some user code makes assumptions about it.

[sbc100]

Isn't such user code broken/non-portable then?

[hoodmane]

Yes.

[sbc100]

I suggest we close this as WAI then.

[sbc100]

If we feel strongly we can open an emscripten but to change time_t to be 64-bit.. assuming i remember correctly and it doesn't effect the syscall layer it should be an easy change.

[hoodmane]

Sounds reasonable to me. We do see similar issues with e.g. function pointer casting etc where the user code is not portable in some way but when possible it is easier to work around it in one place rather than fixing the non portable packages one at a time.

[samuelcolvin]

I'm confused, maybe I'm missing something?

datetime.fromtimestamp(-2147483649) is not really user code, and it's not broken.

[hoodmane]

Well the issue is that the argument to datetime.fromtimestamp has to fit inside size_t or else Python raises an error:
https://github.com/python/cpython/blob/3.10/Python/pytime.c#L73

So in order to use datetime.fromtimestamp in a way that is portable to 32 bit platforms, it is necessary to check somehow check how big size_t is and only pass timestamps that fit in size_t. I guess you'd need to do:

import sysconfig
time_t_size = sysconfig.get_config_var("SIZEOF_TIME_T")
max_time = (1 <<  (time_t_size - 1)) - 1
min_time = - max_time + 1

[hoodmane]

Ideally it would be nice to fix this because the fewer things that are different in Pyodide than in normal Python, the less time people have to waste learning about the differences.

[sbc100]

I'm confused, maybe I'm missing something?

datetime.fromtimestamp(-2147483649) is not really user code, and it's not broken.

Its not broken, but its non-portable in the sense that that it is assuming that size of the size_t. Raising an exception when it doesn't, is, IIUC, not a bug in python, but just the expect behaviour of python.