Skip to content

GH1432 Partial resolution for some TODOs #1492

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+33 −8
Open

GH1432 Partial resolution for some TODOs #1492

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
367ee7d
GH1432 Partial resolution
loicdiridollou Nov 15, 2025
f12e0fe
GH1432 PR Feedback
loicdiridollou Nov 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions pandas-stubs/_libs/tslibs/timestamps.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ from pandas._libs.tslibs import (
Tick,
Timedelta,
)
from pandas._libs.tslibs.nattype import NaTType
from pandas._typing import (
PeriodFrequency,
ShapeT,
Expand Down Expand Up @@ -228,7 +229,7 @@ class Timestamp(datetime, SupportsIndex):
def __radd__(
self, other: np_ndarray[ShapeT, np.timedelta64]
) -> np_ndarray[ShapeT, np.datetime64]: ...
# TODO: pandas-dev/pandas-stubs#1432 test dt64
def __rsub__(self, other: datetime | np.datetime64) -> Timedelta: ...
@overload # type: ignore[override]
def __sub__(self, other: datetime | np.datetime64) -> Timedelta: ...
@overload
Expand Down Expand Up @@ -284,7 +285,15 @@ class Timestamp(datetime, SupportsIndex):
@property
def asm8(self) -> np.datetime64: ...
def tz_convert(self, tz: TimeZones) -> Self: ...
# TODO: pandas-dev/pandas-stubs#1432 could return NaT?
@overload
def tz_localize( # type: ignore[overload-overlap] # pyright: ignore[reportOverlappingOverload]
self,
tz: TimeZones,
ambiguous: _Ambiguous = "raise",
*,
nonexistent: Literal["NaT"],
) -> Self | NaTType: ...
@overload
def tz_localize(
self,
tz: TimeZones,
Expand Down
4 changes: 2 additions & 2 deletions pandas-stubs/core/indexes/interval.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,8 +322,8 @@ def interval_range(
) -> IntervalIndex[Interval[pd.Timestamp]]: ...
@overload
def interval_range(
*,
start: None = None,
*,
end: _TimestampLike,
periods: int | None = ...,
freq: Frequency | dt.timedelta | None = ...,
Expand All @@ -341,8 +341,8 @@ def interval_range(
) -> IntervalIndex[Interval[pd.Timedelta]]: ...
@overload
def interval_range(
*,
start: None = None,
*,
end: _TimedeltaLike,
periods: int | None = ...,
freq: Frequency | dt.timedelta | None = ...,
Expand Down
6 changes: 3 additions & 3 deletions pandas-stubs/io/orc.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
from typing import Any

from fsspec.spec import AbstractFileSystem # pyright: ignore[reportMissingTypeStubs]
from pandas import DataFrame
from pyarrow.fs import FileSystem

from pandas._libs.lib import _NoDefaultDoNotUse
from pandas._typing import (
Expand All @@ -14,8 +16,6 @@ def read_orc(
path: FilePath | ReadBuffer[bytes],
columns: list[HashableT] | None = None,
dtype_backend: DtypeBackend | _NoDefaultDoNotUse = "numpy_nullable",
Copy link
Contributor

@cmp0xff cmp0xff Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 3.0 the default value will be switched from "numpy_nullable" to no_default. I think this can be confusing to users using the stable releases 2.x. What should we do @Dr-Irv ? Anyway, we can address this in a separate PR.

Copy link
Collaborator

@Dr-Irv Dr-Irv Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the 2.3 default for now, and create an issue that we need to fix it when 3.0 is released.

I still think there are things in the release notes for 2.0, 2.1, 2.2 and 2.3 that we never put in the stubs. I created a 2.0 tracker, and I think many of those are still open. Never had the time to create trackers for 2.1, 2.2 and 2.3. And when 3.0 is released, we should create a 3.0 tracker. So we could wait until then instead of creating the issue now

# TODO: pandas-dev/pandas-stubs#1432 type with the correct pyarrow types
# filesystem: pyarrow.fs.FileSystem | fsspec.spec.AbstractFileSystem
filesystem: Any | None = None,
filesystem: FileSystem | AbstractFileSystem | None = None,
**kwargs: Any,
) -> DataFrame: ...
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ beautifulsoup4 = ">=4.14.2"
html5lib = ">=1.1"
python-calamine = ">=0.2.0"
pyarrow-stubs = { version = ">=20.0.0.20250928", python = "<4" }
fsspec = "^2025.10.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
Expand Down
13 changes: 12 additions & 1 deletion tests/scalars/test_scalars.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1385,9 +1385,20 @@ def test_timestamp_misc_methods() -> None:
pd.Timestamp,
)
check(
assert_type(ts.tz_localize("US/Pacific", nonexistent="NaT"), pd.Timestamp),
assert_type(
ts.tz_localize("US/Pacific", nonexistent="NaT"), pd.Timestamp | NaTType
),
pd.Timestamp,
)
check(
assert_type(
pd.Timestamp(2025, 3, 9, 2, 30, 0).tz_localize(
"US/Eastern", nonexistent="NaT"
),
pd.Timestamp | NaTType,
),
NaTType,
)
check(
assert_type(ts.tz_localize("US/Pacific", nonexistent="raise"), pd.Timestamp),
pd.Timestamp,
Expand Down
4 changes: 4 additions & 0 deletions tests/test_timefuncs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,9 +95,13 @@ def test_types_init() -> None:
def test_types_arithmetic() -> None:
ts = pd.to_datetime("2021-03-01")
ts2 = pd.to_datetime("2021-01-01")
ts_np = np.datetime64("2021-01-01")
delta = pd.to_timedelta("1 day")

check(assert_type(ts - ts2, pd.Timedelta), pd.Timedelta)
check(assert_type(ts - ts_np, pd.Timedelta), pd.Timedelta)
# TODO: pandas-dev/pandas-stubs#1432 mypy sees datetime.timedelta but pyright is correct
# check(assert_type(ts_np - ts, pd.Timedelta), pd.Timedelta)
Comment on lines +103 to +104
Copy link
Contributor

@cmp0xff cmp0xff Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see our __rsub__ is correct but mypy takes ts_np.__sub__ as a higher priority. This has happened a lot in Series and Index arithmetic. I think neither mypy nor pyright is correct, we should see Any here because of the discrepency. Do you have any suggestion to improve here, @Dr-Irv ?

Copy link
Collaborator

@Dr-Irv Dr-Irv Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit of a mess, because I don't think that numpy can know whether it supports subtraction or not with a datetime:

>>> import numpy as np
>>> import datetime as dt
>>> ts_np = np.datetime64("2021-01-01")
>>> ts_dt = dt.datetime(2020,12,1)
>>> ts_np - ts_dt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for -: 'datetime.date' and 'datetime.datetime'
>>> ts_dt = dt.datetime(2020,12,1,3,24)
>>> ts_np - ts_dt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for -: 'datetime.date' and 'datetime.datetime'
>>> ts_np = np.datetime64("2021-01-01T03:24")
>>>
>>> ts_np - ts_dt
datetime.timedelta(days=31)
>>> ts_dt = dt.datetime(2021,12,1)
>>> ts_np - ts_dt
datetime.timedelta(days=-334, seconds=12240)

So if the numpy datetime64 does not specify hour/minute, then you can't do subtraction. But if it does have the hour/minute, you can. And their typing can't see the difference, which makes sense.

I suggest doing tests with a numpy type that has no hour/minute resolution (as in this PR) AND with hour/minute resolution.

Without the resolution, it will fail at runtime, but we can't detect it, so put it in if TYPE_CHECKING_INVALID_USAGE with the ignores that detect it (in this case, need a pyright ignore, but not a mypy one)

With the resolution, it will work at runtime, but we can't get the right type with either type checker (pyright will reject, mypy will fail in assert type), so put in appropriate ignores.

In both cases, include comments as to what is going on.

Note - I think pyright is wrong here in not having numpy type declarations figure out that np.datetime64.__sub__(pd.Timestamp) is valid from a type perspective, because pd.Timestamp is a subclass of datetime.datetime But the __sub__() in numpy will take precedence here because it has to allow __sub__(datetime.datetime)

Created a pyright issue: microsoft/pyright#11135

check(assert_type(ts + delta, pd.Timestamp), pd.Timestamp)
check(assert_type(ts - delta, pd.Timestamp), pd.Timestamp)
check(assert_type(ts - dt.datetime(2021, 1, 3), pd.Timedelta), pd.Timedelta)
Expand Down