Python type checking return list base model

[Andrew-Chen-Wang]

Hi we're utilizing a helper class to parse over the paginated lists as so:

from typing import Any, Callable, ParamSpec, TypeVar

from merge.client import Merge
from merge.core import ApiError
from merge.resources.hris import Employee, Group

_T = ParamSpec("_T")
_R = TypeVar("_R", covariant=True)


class Middleware:
    def __init__(self, merge_client: Merge):
        self.merge_client = merge_client

    def _executor(self, func: Callable[_T, _R], *args: _T.args, **kwargs: _T.kwargs) -> list[Any]:
        """Exhausts a paginated API until all data is retrieved"""
        data = []

        cursor = ""
        while cursor is not None:
            try:
                next_page = func(*args, **kwargs, cursor=cursor)  # type: ignore[arg-type]

                for result in next_page.results:  # type: ignore[attr-defined]
                    data.append(result)

                cursor = next_page.next  # type: ignore[attr-defined]
            except Exception as e:
                logger.error(
                    f"Error while fetching users from Merge. Breaking pagination early: {e}"
                )
                break

        return data

    def get_users(self) -> list[Employee]:
        return self._executor(self.merge_client.hris.employees.list)

Noticing the mypy type ignores, we're having issues with the ParamSpec variable and the return type generic TypeVar covariant. It would be great if all return list return types could subclass from a base Pydantic model as all paginated pydantic models shown are like:

class PaginatedEmployeeList(pydantic.BaseModel):
    next: typing.Optional[str]
    previous: typing.Optional[str]
    results: typing.Optional[typing.List[Employee]]

where the only difference is the "Employee" model which could easily be made a template variable. This way, we can make the return type the base pagination list Pydantic model instead of a generic TypeVar and remove the type ignores.

Thanks:)

[lucasgadams]

Agreed, i was just looking into doing the same thing and then saw that there is no paginated base class which makes the typing difficult.

[dsinghvi]

@lucasgadams thanks for filing the issue, currently the Merge team is not planning to update their OpenAPI spec to share a generic paginated class. If i were you, I'd build this helper without typing since you can rely on the fact that the classes have similar attributes.