smart_deepcopy causes sqlalchemy boolean value error on import

[coneybeare]

Checks

• I added a descriptive title to this issue
• I have searched (google, github) for similar issues and couldn't find anything
• I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

root@20116b25f897:/[REDACTED] # python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.8.2
            pydantic compiled: False
                 install path: /usr/local/lib/python3.10/site-packages/pydantic
               python version: 3.10.5 (main, Jun  7 2022, 18:49:47) [GCC 10.2.1 20210110]
                     platform: Linux-5.10.104-linuxkit-x86_64-with-glibc2.31
     optional deps. installed: ['dotenv', 'email-validator', 'typing-extensions']

"""Base class for all database models."""

from typing import Any, Dict, Optional
from uuid import UUID, uuid4

import sqlalchemy as sa
import sqlalchemy.dialects.postgresql as pg
from inflection import underscore
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import declared_attr
from sqlalchemy_utils import Timestamp

from pydantic import BaseModel as _BaseModel

class CommonBase(_BaseModel, Timestamp):
    """SQLAlchemy Base common base class for all database models."""

    id: UUID = sa.Column(pg.UUID(as_uuid=True), primary_key=True, default=uuid4)

    @declared_attr
    def __tablename__(cls) -> Optional[str]:  # pylint: disable=no-self-argument
        """Define table name for all models as the snake case of the model's name."""
        return underscore(cls.__name__)

    def as_dict(self) -> Dict[str, Any]:
        """Return python dictionary of SQLAlchemy model."""
        return self.__dict__

BaseModel = declarative_base(cls=CommonBase)

On import of this BaseModel, we are getting a hard to track down error that seems to be related to sqlalchemy boolean expressions, but stemming from the smart_deepcopy line in pydantic and I am not quite sure what the real issue is, and whether it is caused by pydantic, sqlalchemy, or my setup (most likely).

The stack trace received on import is:

app/models/base_model.py:15: in <module>
    class CommonBase(_BaseModel, Timestamp):
/usr/local/lib/python3.10/site-packages/pydantic/main.py:299: in __new__
    fields[ann_name] = ModelField.infer(
/usr/local/lib/python3.10/site-packages/pydantic/fields.py:411: in infer
    return cls(
/usr/local/lib/python3.10/site-packages/pydantic/fields.py:342: in __init__
    self.prepare()
/usr/local/lib/python3.10/site-packages/pydantic/fields.py:445: in prepare
    self._set_default_and_type()
/usr/local/lib/python3.10/site-packages/pydantic/fields.py:473: in _set_default_and_type
    default_value = self.get_default()
/usr/local/lib/python3.10/site-packages/pydantic/fields.py:345: in get_default
    return smart_deepcopy(self.default) if self.default_factory is None else self.default_factory()
/usr/local/lib/python3.10/site-packages/pydantic/utils.py:627: in smart_deepcopy
    elif not obj and obj_type in BUILTIN_COLLECTIONS:
/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py:582: in __bool__
    raise TypeError("Boolean value of this clause is not defined")
E   TypeError: Boolean value of this clause is not defined

I placed a debugger just before the raise and found that:

(Pdb) self
Column(None, UUID(as_uuid=True), table=None, primary_key=True, nullable=False, default=ColumnDefault(<function uuid4 at 0x7f99d085e200>))
(Pdb) self.__dict__
{'key': None, 'name': None, 'table': None, 'type': UUID(as_uuid=True), 'is_literal': False, 'primary_key': True, '_user_defined_nullable': symbol('NULL_UNSPECIFIED'), 'nullable': False, 'default': ColumnDefault(<function uuid4 at 0x7f99d085e200>), 'server_default': None, 'server_onupdate': None, 'index': None, 'unique': None, 'system': False, 'doc': None, 'onupdate': None, 'autoincrement': 'auto', 'constraints': set(), 'foreign_keys': set(), 'comment': None, 'computed': None, 'identity': None, '_creation_order': 88, 'comparator': <sqlalchemy.sql.type_api.TypeEngine.Comparator object at 0x7f99d07f0a00>, '_memoized_keys': frozenset({'comparator'})}

This is not new code in pydantic, and it also seems this check in sqlalchemy was added almost a decade ago so I am sure that is must be something wrong with either my expectatins or setup here. While investigating, it seems that this indeed is the correct way to setup a postgres UUID pk column within a pydantic model backed by sqlalchemy, but I am clearly missing something.

Thanks for the incredible work you do on this library!

[samuelcolvin]

Off the top of my head, I don't know.

That's a config setting called copy_on_model_validation try setting it to false and seeing if that avoids a copy that causes the problem.

[coneybeare]

copy_on_model_validation didn't change things. I whittled the setup down to the minimum reproducible case:

from uuid import UUID, uuid4
import sqlalchemy as sa
import sqlalchemy.dialects.postgresql as pg
from pydantic import BaseModel as _BaseModel
from sqlalchemy.ext.declarative import declarative_base

class CommonBase(_BaseModel):

    id: UUID = sa.Column(pg.UUID(as_uuid=True), primary_key=True, default=uuid4)

    class Config:
        copy_on_model_validation = False

BaseModel = declarative_base(cls=CommonBase)

I also updated to pydantic 1.9.1 without change 🤔

[samuelcolvin]

1.9.1 hasn't changed smart_deepcopy so that wouldn't make a difference.

Really I don't think it's correct for the sqlalchemy type to be raising an error on __bool__, but I get why they want to.

I think the best solution would be to catch errors like this in smart_deepcopy and just fall back to using deepcopy, but I wonder whether even deep copy can cope with this type?

Something like

def smart_deepcopy(obj: Obj) -> Obj:
    """
    Return type as is for immutable built-in types
    Use obj.copy() for built-in empty collections
    Use copy.deepcopy() for non-empty collections and unknown objects
    """

    obj_type = obj.__class__
    if obj_type in IMMUTABLE_NON_COLLECTIONS_TYPES:
        return obj  # fastest case: obj is immutable and not collection therefore will not be copied anyway
    try:
        if not obj and obj_type in BUILTIN_COLLECTIONS:
            # faster way for empty collections, no need to copy its members
            return obj if obj_type is tuple else obj.copy()  # type: ignore  # tuple doesn't have copy method
    except (TypeError, ValueError, RuntimeError):
        # do we really dare to catch ALL errors? Seems a bit risky
        pass
    return deepcopy(obj)  # slowest way when we actually might need a deepcopy

What do you think?

PR welcome, but I'm not sure when I'll get around to it - I'm currently flat out on pydantic-core getting ready for pydantic v2.

But at least your issue has made me think about how we deal with deepcopy in v2 😄 .

[coneybeare]

per their docs

Code that wants to check for the presence of a ClauseElement expression should instead say:
if expression is not None:
print("the expression is:", expression)

So could this perhaps be fixed by changing the smart deep copy line to

if obj is None and obj_type in BUILTIN_COLLECTIONS:

Does this method expect other obj references that are falsy rather than None? if so, perhaps this:

if (obj is None or not obj) and obj_type in BUILTIN_COLLECTIONS:

[samuelcolvin]

That doesn't work, None is already covered by the IMMUTABLE_NON_COLLECTIONS_TYPES check above, and this line is using falsy as a proxy for empty on iterable types, so a None check is different.

[coneybeare]

Isn't the immutable collection type check covering None on the obj_type, not obj? Isn't the raise happening on the not obj statement where sqlalchemy added the __bool__ override?

[samuelcolvin]

I'm not sure what you mean, but neither of your code suggestions will work. Best to try it and see what happens.

[coneybeare]

You were right, my train of thought would have required a quite ugly double negative to work. I was only trying to avoid catching errors, but I suspect your proposed approach is the cleanest and safest. Will submit a pr

[ubernostrum]

I believe this is also the source of this issue I just filed against Starlite: litestar-org/litestar#333

There, the default behavior of copy.deepcopy() in Pydantic 1.9.1 is attempting to copy a Redis client instance passed to Starlite's caching config, and failing because that's a non-pickle-able object.

The issue filed at Starlite has a minimal reproduction of attempting to instantiate a starlite.Starlite with a starlite.CacheConfig containing a Redis client instance, and reliably fails on Pydantic 1.9.1 but not Pydantic 1.9.0.