Typing support for both "modes" of single-function hybrid_method/hybrid_property

[Muzer]

Ensure stubs packages are not installed

• No sqlalchemy stub packages is installed (both sqlalchemy-stubs and sqlalchemy2-stubs are not compatible with v2)

Verify if the api is typed

• The api is not in a module listed in #6810 so it should pass type checking

Describe the typing issue

In the project I'm working on I have decided we should write some helper functions for hybrid methods/properties to allow us to, wherever possible, write them as single implementations. This is to avoid potential implementation drift between the Python and SQL Expression versions of these implementations.

However, the advertised way in sqlalchemy to implement one hybrid method that covers both cases is kinda hacky. As far as I can tell you are basically writing it as a method, but having the same function also run as a classmethod with the model type passed instead of a model instance when you want to run in Expression mode.

This obviously means that while it might work fine, when implementing a hybrid_method/hybrid_property as a single function, the expression variant remains completely un-type checked.

I was wondering if there is any way of supporting type checking of both variants somehow? Is this even possible? Maybe the answer is explicitly typing the first argument (ie what would normally be self/cls) and having overloads. Any suggestions from those who have done it?

I'm mostly filing this as an issue as I think if it's currently possible, the docs should make clear how to do it; and if it's not currently possible, but it is something that could potentially be implemented, it'd be nice to see it.

To Reproduce

from datetime import datetime
from typing import overload

from sqlalchemy import ColumnElement, SQLColumnExpression
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column


class SomeModel(DeclarativeBase):
    ID: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)
    archived_at: Mapped[datetime | None] = mapped_column()

    @hybrid_property
    def archived(self) -> bool:
        # This call will ONLY be type-checked in the Python case, not the SQL Expression case!
        return is_not_none(self, self.archived_at)


# This overload will appear to be unused even though it's not!
@overload
def is_not_none[
    T
](magic: type[DeclarativeBase], field: SQLColumnExpression[T | None]) -> ColumnElement[
    bool
]:
    ...


@overload
def is_not_none[T](magic: DeclarativeBase, field: T | None) -> bool:
    ...


def is_not_none[
    T
](
    # We pass this magic value to provide a canonical way of determining if we're being
    # called from the instance method (Python) or the class method (SQL).
    magic: type[DeclarativeBase] | DeclarativeBase,
    # There is a weird mypy bug where if you add `SQLColumnExpression[T | None]` to the
    # below it will fail. Since T can be `SQLColumnExpression[U | None]` anyway leaving
    # it off is fine.
    field: T | None,
) -> (bool | ColumnElement[bool]):
    if isinstance(magic, DeclarativeBase):
        if isinstance(field, SQLColumnExpression):
            raise ValueError("field must not be of type SQLColumnExpression")
        return field is not None
    else:
        if not isinstance(field, SQLColumnExpression):
            raise ValueError("field must be of type SQLColumnExpression")
        return field.is_not(None)

Error

No error, the issue is just that one substantial code path is not being type checked.

Versions

• OS: Linux
• Python: 3.12.3
• SQLAlchemy: 2.0.44
• Type checker (eg: mypy 0.991, pyright 1.1.290, etc): mypy 1.19.1

Additional context

No response

[zzzeek]

hi

ive converted to discussion since this issue is basically "the default way hybrids work can leave holes in typing", there's no "bug" to fix here, typing is typing. meaning, if the code that you want the type checker to look at doesnt exist, it can't look at it :)

even though hybrids started as this neato way to have a single method double as an instance-level, class-level method at the same time, there's no requirement whatsoever that you have to use it that way, you are free to implement @expression as well for explicitness. if you want the expression version to be typed checked, it has to be explicit:

class SomeModel(DeclarativeBase):
    ID: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)
    archived_at: Mapped[datetime | None] = mapped_column()

    @hybrid_property
    def archived(self) -> bool:
        return is_not_none(self, self.archived_at)

    @archived.inplace.expression
    @classmethod
    def _archived(cls) -> ColumnElement[bool]:
        return is_not_none(cls, cls.archived_at)

this follows the example given at https://docs.sqlalchemy.org/en/20/orm/extensions/hybrid.html#using-inplace-to-create-pep-484-compliant-hybrid-properties

now is the request, "can we add a doc note to that section clarifying that these methods may be required for typing support" ? that's a doc issue sure, though not a typing bug.

I have decided we should write some helper functions for hybrid methods/properties to allow us to, wherever possible, write them as single implementations.

However, the advertised way in sqlalchemy to implement one hybrid method that covers both cases is kinda hacky.

The critique "kinda hacky" suggests there's some other way to write a method that works as self- or cls- at the same time, without writing two methods, that is not "kinda hacky". Python is pretty strict about "this method is an instance method, that method is a classmethod". There's no explicit concept I'm aware of which implicitly allows "it's both", you can of course put the implementation in some common function but one method needs to have @classmethod for it to see classmethod, full stop. the typed approach I have above would be the version that's not "hacky" to me.

[zzzeek]

The whole point of the exercise from my perspective is trying to reduce repetition at the model level, because there's always potential for two implementations to get out-of-sync i

yeah that's why we thought hybrids were cool when we first came up with them!

So I was basically wondering if it might be possible to reduce that boilerplate somehow.

this is all running up against python's typing pep-484 which was invented many years after our hybrid feature. it pretty much has zero tolerance for "tricks" that save on verbosity and repetition. We've learned to embrace things like TYPE_CHECKING blocks that can give type checkers the cues they need while not having to add more runtime code though that doesn't really provide any "out" here.

looking at your example again it looks like your "implementation" is taking the "hybrid" "self/cls" input and then doing an isinstance to break out again into two implementations. This is actually the opposite of how I would implement this; the reason we have methods in OO and also the reason hybrids have @hybrid and @hybrid.expression is so that different implementations corresponding to different object types are dispatched automatically. that is, using one of those academic OO examples, one would not do this:

# anti-pattern; re-invent dispatch

class Vehicle:
    def number_of_wheels(self):
        if isinstance(self, Car):
            return 4
        elif isinstance(self, Motorcycle):
            return 2
        else:
            raise ValueError("expected Car or Motorcycle")

class Car(Vehicle):
    pass

class Motorcycle(Vehicle):
    pass

right? that would be silly. the whole point of Car/Motorcycle is to put methods specific to those types on each class, Python does the dispatch (the "which method do we call for which type?") for you:

# correct way

class Vehicle:
    def number_of_wheels(self):
        raise NotImplementedError()

class Car(Vehicle):
    def number_of_wheels(self):
       return 4

class Motorcycle(Vehicle):
     def number_of_wheels(self):
        return 2

the use of @hybrid / @hybrid.expression is similar; two methods that give you the place to do the thing you need without runtime isinstance() checks. In this case, the other thing it looks like you're trying to get is a reusable "is_not_none", so here's one way to do that using @hybrid_method:

from datetime import datetime
from typing import Any

from sqlalchemy import ColumnElement
from sqlalchemy import SQLColumnExpression
from sqlalchemy.ext.hybrid import hybrid_method
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.orm import DeclarativeBase
from sqlalchemy.orm import Mapped
from sqlalchemy.orm import mapped_column


class IsNotNoneMixin:
    @hybrid_method
    def is_not_none(self, field: Any) -> bool:
        return field is not None

    @is_not_none.expression
    @classmethod
    def _is_not_none(cls, field: SQLColumnExpression[Any]) -> ColumnElement[bool]:
        return field.is_not(None)

class Base(DeclarativeBase):
    pass

class SomeModel(Base, IsNotNoneMixin):
    __tablename__ = "some_table"

    ID: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)
    archived_at: Mapped[datetime | None] = mapped_column()

    @hybrid_property
    def archived(self) -> bool:
        return self.is_not_none(self.archived_at)



if __name__ == "__main__":
    bool_col_expr: SQLColumnExpression[bool] = SomeModel.archived

    sm = SomeModel(archived_at=datetime.today())

    bool_expr: bool = sm.archived

notes on above:

1. I converted to use a real model with tablename etc. so we get a real result at the end
2. you can use @is_not_none.inplace.expression with hybrid method as well, it's there, however it's not necessary whereas it is with @hybrid_property when changing the name. however, there seems to be a mypy bug that is rejecting the use of @is_not_none.inplace.expression that we likely need to fix on our end (so there is a bug)

using the above form, you:

1. dont need isinstance()
2. dont need @overload
3. dont need to raise or test for ValueError

[zzzeek]

ive uncovered some less than ideal aspects of hybrid_method for typing over at #13253 . also you want both implementations to be met. the most canonical way to do this as written is like this:

class SomeModel(Base):
    __tablename__ = "some_table"

    ID: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)
    archived_at: Mapped[datetime | None] = mapped_column()

    @hybrid_property
    def archived(self) -> bool:
        return self.archived_at is not None

    @archived.inplace.expression
    @classmethod
    def _archived_expr(cls) -> SQLColumnExpression[bool]:
        return cls.archived_at.is_not(None)

overall you need to have two separate implementation methods no matter what.

[Muzer]

I like your example with the mixin, that definitely gets around the isinstance nastiness and I'll definitely have a play with that! I hadn't thought about using further hybrid_methods to avoid the need for overload nastiness.

But it still has the same problem I brought up in the first place - lets say you had a hybrid method which used, say, five of these mixins in quite a complex way. The way you've shown above would still only type-check the archived method in the Python case, not the SQL Expression case. Simplifying further say we had:

@hybrid_method
def do_some_complex_thing(self) -> bool:
    return self.is_not_none(self.some_field) & self.is_not_none(True) # oops, doesn't work in the SQL case!

^ This would then not throw a type error even though it would have done if we'd explicitly written out the expression variant! That's the inconsistency I'd like to find a way to solve, if at all possible.

[zzzeek]

there's no solution. the point of hybrids is to treat "self" and "cls" through common pipelines. pep484 has zero notion of that. so you need to either not send self/cls through common pipelines or you need to rely on real unit tests (prefer the latter).

[CaselIT]

I wonder if we could do that by having a normal method and manually calling the decorator?

Like this should work

def my_method(thing: type[X]|X):
  ...

class X(Base):
  ...
  prop = hybrid_property(my_method)

it's not really ergonomic or anything pythonic at all, but it should type check fine