hypothesis[django]: CharField causes strange assertion issue

[dakotahorstman]

I'm encountering an issue with hypothesis that I originally thought was due to compatibility issues between pytest-django and hypothesis; however, after running a simple test case with unittest and hypothesis, I believe this issue is being caused by hypothesis. Curiously, this issue occurs on just the CharField on my model and not the BooleanField.

Below, I provide an MRE that can be installed on the most basic django app. If there are any further details I can provide, let me know. Testing is executed with poetry run coverage run manage.py test.

Environment

docker: dev-container, vscode
python: 3.11.6
django: 4.2.7
hypothesis[django]: 6.90.0

Full environment

alabaster==0.7.13 ; python_version >= "3.11" and python_version < "3.12"
amqp==5.2.0 ; python_version >= "3.11" and python_version < "3.12"
anyascii==0.3.2 ; python_version >= "3.11" and python_version < "3.12"
anyio==4.1.0 ; python_version >= "3.11" and python_version < "3.12"
argon2-cffi-bindings==21.2.0 ; python_version >= "3.11" and python_version < "3.12"
argon2-cffi==23.1.0 ; python_version >= "3.11" and python_version < "3.12"
asgiref==3.7.2 ; python_version >= "3.11" and python_version < "3.12"
async-timeout==4.0.3 ; python_version >= "3.11" and python_full_version <= "3.11.2"
attrs==23.1.0 ; python_version >= "3.11" and python_version < "3.12"
autobahn==23.6.2 ; python_version >= "3.11" and python_version < "3.12"
automat==22.10.0 ; python_version >= "3.11" and python_version < "3.12"
babel==2.13.1 ; python_version >= "3.11" and python_version < "3.12"
beautifulsoup4==4.11.2 ; python_version >= "3.11" and python_version < "3.12"
billiard==4.2.0 ; python_version >= "3.11" and python_version < "3.12"
blinker==1.7.0 ; python_version >= "3.11" and python_version < "3.12"
brotli==1.1.0 ; python_version >= "3.11" and python_version < "3.12"
cachetools==5.3.2 ; python_version >= "3.11" and python_version < "3.12"
celery==5.3.6 ; python_version >= "3.11" and python_version < "3.12"
certifi==2023.11.17 ; python_version >= "3.11" and python_version < "3.12"
cffi==1.16.0 ; python_version >= "3.11" and python_version < "3.12"
cfgv==3.4.0 ; python_version >= "3.11" and python_version < "3.12"
channels-redis==4.1.0 ; python_version >= "3.11" and python_version < "3.12"
channels==4.0.0 ; python_version >= "3.11" and python_version < "3.12"
channels[daphne]==4.0.0 ; python_version >= "3.11" and python_version < "3.12"
chardet==5.2.0 ; python_version >= "3.11" and python_version < "3.12"
charset-normalizer==3.3.2 ; python_version >= "3.11" and python_version < "3.12"
click-didyoumean==0.3.0 ; python_version >= "3.11" and python_version < "3.12"
click-plugins==1.1.1 ; python_version >= "3.11" and python_version < "3.12"
click-repl==0.3.0 ; python_version >= "3.11" and python_version < "3.12"
click==8.1.7 ; python_version >= "3.11" and python_version < "3.12"
colorama==0.4.6 ; python_version >= "3.11" and python_version < "3.12"
configargparse==1.7 ; python_version >= "3.11" and python_version < "3.12"
constantly==23.10.4 ; python_version >= "3.11" and python_version < "3.12"
coverage==7.3.2 ; python_version >= "3.11" and python_version < "3.12"
coverage[toml]==7.3.2 ; python_version >= "3.11" and python_version < "3.12"
crispy-bootstrap5==2023.10 ; python_version >= "3.11" and python_version < "3.12"
cron-descriptor==1.4.0 ; python_version >= "3.11" and python_version < "3.12"
cryptography==41.0.5 ; python_version >= "3.11" and python_version < "3.12"
cssbeautifier==1.14.11 ; python_version >= "3.11" and python_version < "3.12"
daphne==4.0.0 ; python_version >= "3.11" and python_version < "3.12"
defusedxml==0.7.1 ; python_version >= "3.11" and python_version < "3.12"
detect-secrets==1.4.0 ; python_version >= "3.11" and python_version < "3.12"
distlib==0.3.7 ; python_version >= "3.11" and python_version < "3.12"
django-allauth==0.58.2 ; python_version >= "3.11" and python_version < "3.12"
django-appconf==1.0.6 ; python_version >= "3.11" and python_version < "3.12"
django-celery-beat==2.5.0 ; python_version >= "3.11" and python_version < "3.12"
django-compressor==4.4 ; python_version >= "3.11" and python_version < "3.12"
django-cors-headers==4.3.1 ; python_version >= "3.11" and python_version < "3.12"
django-coverage-plugin==3.1.0 ; python_version >= "3.11" and python_version < "3.12"
django-crispy-forms==2.1 ; python_version >= "3.11" and python_version < "3.12"
django-debug-toolbar==4.2.0 ; python_version >= "3.11" and python_version < "3.12"
django-environ==0.11.2 ; python_version >= "3.11" and python_version < "3.12"
django-extensions==3.2.3 ; python_version >= "3.11" and python_version < "3.12"
django-filter==23.4 ; python_version >= "3.11" and python_version < "3.12"
django-model-utils==4.3.1 ; python_version >= "3.11" and python_version < "3.12"
django-modelcluster==6.1 ; python_version >= "3.11" and python_version < "3.12"
django-permissionedforms==0.1 ; python_version >= "3.11" and python_version < "3.12"
django-redis==5.4.0 ; python_version >= "3.11" and python_version < "3.12"
django-stubs-ext==4.2.5 ; python_version >= "3.11" and python_version < "3.12"
django-stubs[compatible-mypy]==4.2.6 ; python_version >= "3.11" and python_version < "3.12"
django-taggit==4.0.0 ; python_version >= "3.11" and python_version < "3.12"
django-timezone-field==6.1.0 ; python_version >= "3.11" and python_version < "3.12"
django-treebeard==4.7 ; python_version >= "3.11" and python_version < "3.12"
django-upgrade==1.15.0 ; python_version >= "3.11" and python_version < "3.12"
django==4.2.7 ; python_version >= "3.11" and python_version < "3.12"
djangorestframework==3.14.0 ; python_version >= "3.11" and python_version < "3.12"
djlint==1.34.0 ; python_version >= "3.11" and python_version < "3.12"
docutils==0.20.1 ; python_version >= "3.11" and python_version < "3.12"
draftjs-exporter==2.1.7 ; python_version >= "3.11" and python_version < "3.12"
drf-spectacular==0.26.5 ; python_version >= "3.11" and python_version < "3.12"
editorconfig==0.12.3 ; python_version >= "3.11" and python_version < "3.12"
et-xmlfile==1.1.0 ; python_version >= "3.11" and python_version < "3.12"
execnet==2.0.2 ; python_version >= "3.11" and python_version < "3.12"
factory-boy==3.3.0 ; python_version >= "3.11" and python_version < "3.12"
faker==20.1.0 ; python_version >= "3.11" and python_version < "3.12"
filelock==3.13.1 ; python_version >= "3.11" and python_version < "3.12"
filetype==1.2.0 ; python_version >= "3.11" and python_version < "3.12"
flask-basicauth==0.2.0 ; python_version >= "3.11" and python_version < "3.12"
flask-cors==4.0.0 ; python_version >= "3.11" and python_version < "3.12"
flask==3.0.0 ; python_version >= "3.11" and python_version < "3.12"
flower==2.0.1 ; python_version >= "3.11" and python_version < "3.12"
furo==2023.9.10 ; python_version >= "3.11" and python_version < "3.12"
gevent==23.9.1 ; python_version >= "3.11" and python_version < "3.12"
geventhttpclient==2.0.11 ; python_version >= "3.11" and python_version < "3.12"
greenlet==3.0.1 ; platform_python_implementation == "CPython" and python_version >= "3.11" and python_version < "3.12"
h11==0.14.0 ; python_version >= "3.11" and python_version < "3.12"
hiredis==2.2.3 ; python_version >= "3.11" and python_version < "3.12"
html-tag-names==0.1.2 ; python_version >= "3.11" and python_version < "3.12"
html-void-elements==0.1.0 ; python_version >= "3.11" and python_version < "3.12"
html5lib==1.1 ; python_version >= "3.11" and python_version < "3.12"
httptools==0.6.1 ; python_version >= "3.11" and python_version < "3.12"
humanize==4.9.0 ; python_version >= "3.11" and python_version < "3.12"
hyperlink==21.0.0 ; python_version >= "3.11" and python_version < "3.12"
hypothesis==6.90.0 ; python_version >= "3.11" and python_version < "3.12"
identify==2.5.32 ; python_version >= "3.11" and python_version < "3.12"
idna==3.6 ; python_version >= "3.11" and python_version < "3.12"
imagesize==1.4.1 ; python_version >= "3.11" and python_version < "3.12"
incremental==22.10.0 ; python_version >= "3.11" and python_version < "3.12"
inflection==0.5.1 ; python_version >= "3.11" and python_version < "3.12"
iniconfig==2.0.0 ; python_version >= "3.11" and python_version < "3.12"
itsdangerous==2.1.2 ; python_version >= "3.11" and python_version < "3.12"
jinja2==3.1.2 ; python_version >= "3.11" and python_version < "3.12"
jsbeautifier==1.14.11 ; python_version >= "3.11" and python_version < "3.12"
json5==0.9.14 ; python_version >= "3.11" and python_version < "3.12"
jsonschema-specifications==2023.11.1 ; python_version >= "3.11" and python_version < "3.12"
jsonschema==4.20.0 ; python_version >= "3.11" and python_version < "3.12"
kombu==5.3.4 ; python_version >= "3.11" and python_version < "3.12"
l18n==2021.3 ; python_version >= "3.11" and python_version < "3.12"
livereload==2.6.3 ; python_version >= "3.11" and python_version < "3.12"
locust==2.19.0 ; python_version >= "3.11" and python_version < "3.12"
markupsafe==2.1.3 ; python_version >= "3.11" and python_version < "3.12"
msgpack==1.0.7 ; python_version >= "3.11" and python_version < "3.12"
mypy-extensions==1.0.0 ; python_version >= "3.11" and python_version < "3.12"
mypy==1.6.1 ; python_version >= "3.11" and python_version < "3.12"
nodeenv==1.8.0 ; python_version >= "3.11" and python_version < "3.12"
oauthlib==3.2.2 ; python_version >= "3.11" and python_version < "3.12"
openpyxl==3.1.2 ; python_version >= "3.11" and python_version < "3.12"
packaging==23.2 ; python_version >= "3.11" and python_version < "3.12"
pathspec==0.11.2 ; python_version >= "3.11" and python_version < "3.12"
pillow-heif==0.13.1 ; python_version >= "3.11" and python_version < "3.12"
pillow==10.1.0 ; python_version >= "3.11" and python_version < "3.12"
platformdirs==4.0.0 ; python_version >= "3.11" and python_version < "3.12"
pluggy==1.3.0 ; python_version >= "3.11" and python_version < "3.12"
pre-commit==3.5.0 ; python_version >= "3.11" and python_version < "3.12"
prettier==0.0.7 ; python_version >= "3.11" and python_version < "3.12"
prometheus-client==0.19.0 ; python_version >= "3.11" and python_version < "3.12"
prompt-toolkit==3.0.41 ; python_version >= "3.11" and python_version < "3.12"
psutil==5.9.6 ; python_version >= "3.11" and python_version < "3.12"
psycopg-c==3.1.13 ; python_version >= "3.11" and python_version < "3.12"
psycopg[c]==3.1.13 ; python_version >= "3.11" and python_version < "3.12"
pyasn1-modules==0.3.0 ; python_version >= "3.11" and python_version < "3.12"
pyasn1==0.5.1 ; python_version >= "3.11" and python_version < "3.12"
pycparser==2.21 ; python_version >= "3.11" and python_version < "3.12"
pygments==2.17.2 ; python_version >= "3.11" and python_version < "3.12"
pyjwt[crypto]==2.8.0 ; python_version >= "3.11" and python_version < "3.12"
pyopenssl==23.3.0 ; python_version >= "3.11" and python_version < "3.12"
pyproject-api==1.6.1 ; python_version >= "3.11" and python_version < "3.12"
pytest-cov==4.1.0 ; python_version >= "3.11" and python_version < "3.12"
pytest-django==4.7.0 ; python_version >= "3.11" and python_version < "3.12"
pytest-factoryboy==2.6.0 ; python_version >= "3.11" and python_version < "3.12"
pytest-mock==3.12.0 ; python_version >= "3.11" and python_version < "3.12"
pytest-xdist==3.5.0 ; python_version >= "3.11" and python_version < "3.12"
pytest==7.4.3 ; python_version >= "3.11" and python_version < "3.12"
python-crontab==3.0.0 ; python_version >= "3.11" and python_version < "3.12"
python-dateutil==2.8.2 ; python_version >= "3.11" and python_version < "3.12"
python-dotenv==1.0.0 ; python_version >= "3.11" and python_version < "3.12"
python-slugify==8.0.1 ; python_version >= "3.11" and python_version < "3.12"
python3-openid==3.2.0 ; python_version >= "3.11" and python_version < "3.12"
pytz==2023.3.post1 ; python_version >= "3.11" and python_version < "3.12"
pywin32==306 ; python_version >= "3.11" and python_version < "3.12" and platform_system == "Windows"
pyyaml==6.0.1 ; python_version >= "3.11" and python_version < "3.12"
pyzmq==25.1.1 ; python_version >= "3.11" and python_version < "3.12"
rcssmin==1.1.1 ; python_version >= "3.11" and python_version < "3.12"
redis==5.0.1 ; python_version >= "3.11" and python_version < "3.12"
referencing==0.31.0 ; python_version >= "3.11" and python_version < "3.12"
regex==2023.10.3 ; python_version >= "3.11" and python_version < "3.12"
requests-oauthlib==1.3.1 ; python_version >= "3.11" and python_version < "3.12"
requests==2.31.0 ; python_version >= "3.11" and python_version < "3.12"
rjsmin==1.2.1 ; python_version >= "3.11" and python_version < "3.12"
roundrobin==0.0.4 ; python_version >= "3.11" and python_version < "3.12"
rpds-py==0.13.1 ; python_version >= "3.11" and python_version < "3.12"
ruff==0.1.6 ; python_version >= "3.11" and python_version < "3.12"
sentry-sdk==1.37.1 ; python_version >= "3.11" and python_version < "3.12"
service-identity==23.1.0 ; python_version >= "3.11" and python_version < "3.12"
setuptools==69.0.2 ; python_version >= "3.11" and python_version < "3.12"
six==1.16.0 ; python_version >= "3.11" and python_version < "3.12"
sniffio==1.3.0 ; python_version >= "3.11" and python_version < "3.12"
snowballstemmer==2.2.0 ; python_version >= "3.11" and python_version < "3.12"
sortedcontainers==2.4.0 ; python_version >= "3.11" and python_version < "3.12"
soupsieve==2.5 ; python_version >= "3.11" and python_version < "3.12"
sphinx-autobuild==2021.3.14 ; python_version >= "3.11" and python_version < "3.12"
sphinx-basic-ng==1.0.0b2 ; python_version >= "3.11" and python_version < "3.12"
sphinx==7.2.6 ; python_version >= "3.11" and python_version < "3.12"
sphinxcontrib-applehelp==1.0.7 ; python_version >= "3.11" and python_version < "3.12"
sphinxcontrib-devhelp==1.0.5 ; python_version >= "3.11" and python_version < "3.12"
sphinxcontrib-htmlhelp==2.0.4 ; python_version >= "3.11" and python_version < "3.12"
sphinxcontrib-jsmath==1.0.1 ; python_version >= "3.11" and python_version < "3.12"
sphinxcontrib-qthelp==1.0.6 ; python_version >= "3.11" and python_version < "3.12"
sphinxcontrib-serializinghtml==1.1.9 ; python_version >= "3.11" and python_version < "3.12"
sqlparse==0.4.4 ; python_version >= "3.11" and python_version < "3.12"
telepath==0.3.1 ; python_version >= "3.11" and python_version < "3.12"
text-unidecode==1.3 ; python_version >= "3.11" and python_version < "3.12"
tokenize-rt==5.2.0 ; python_version >= "3.11" and python_version < "3.12"
tornado==6.3.3 ; python_version >= "3.11" and python_version < "3.12"
tox==4.11.3 ; python_version >= "3.11" and python_version < "3.12"
tqdm==4.66.1 ; python_version >= "3.11" and python_version < "3.12"
twisted-iocpsupport==1.0.4 ; python_version >= "3.11" and python_version < "3.12" and platform_system == "Windows"
twisted[tls]==23.10.0 ; python_version >= "3.11" and python_version < "3.12"
txaio==23.1.1 ; python_version >= "3.11" and python_version < "3.12"
types-pytz==2023.3.1.1 ; python_version >= "3.11" and python_version < "3.12"
types-pyyaml==6.0.12.12 ; python_version >= "3.11" and python_version < "3.12"
typing-extensions==4.8.0 ; python_version >= "3.11" and python_version < "3.12"
tzdata==2023.3 ; python_version >= "3.11" and python_version < "3.12"
uritemplate==4.1.1 ; python_version >= "3.11" and python_version < "3.12"
urllib3==2.1.0 ; python_version >= "3.11" and python_version < "3.12"
uvicorn[standard]==0.24.0.post1 ; python_version >= "3.11" and python_version < "3.12"
uvloop==0.19.0 ; (sys_platform != "win32" and sys_platform != "cygwin") and platform_python_implementation != "PyPy" and python_version >= "3.11" and python_version < "3.12"
vine==5.1.0 ; python_version >= "3.11" and python_version < "3.12"
virtualenv==20.24.7 ; python_version >= "3.11" and python_version < "3.12"
wagtail==5.2.1 ; python_version >= "3.11" and python_version < "3.12"
watchfiles==0.21.0 ; python_version >= "3.11" and python_version < "3.12"
wcwidth==0.2.12 ; python_version >= "3.11" and python_version < "3.12"
webencodings==0.5.1 ; python_version >= "3.11" and python_version < "3.12"
websockets==12.0 ; python_version >= "3.11" and python_version < "3.12"
werkzeug==3.0.1 ; python_version >= "3.11" and python_version < "3.12"
whitenoise==6.6.0 ; python_version >= "3.11" and python_version < "3.12"
willow[heif]==1.6.2 ; python_version >= "3.11" and python_version < "3.12"
zope-event==5.0 ; python_version >= "3.11" and python_version < "3.12"
zope-interface==6.1 ; python_version >= "3.11" and python_version < "3.12"

Error & Traceback

Found 3 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).

F..
======================================================================
FAIL: test_base_tag (wishbone.data.tests.test_models.TestBaseTag.test_base_tag)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/app/wishbone/data/tests/test_models.py", line 17, in test_base_tag
    def test_base_tag(self, instance: DeviceTag) -> None:
               ^^^^^^^
  File "/home/dev-user/.cache/pypoetry/virtualenvs/wishbone-9TtSrW0h-py3.11/lib/python3.11/site-packages/hypothesis/core.py", line 1469, in wrapped_test
    raise the_error_hypothesis_found
  File "/app/wishbone/data/tests/test_models.py", line 20, in test_base_tag
    self.assertIsInstance(instance.name, str)
AssertionError: 'New Tag' is not an instance of <class 'str'>
Falsifying example: test_base_tag(
    self=<wishbone.data.tests.test_models.TestBaseTag testMethod=test_base_tag>,
    instance=<DeviceTag: DeviceTag object (2)>,
)

Test Case

from hypothesis import given
from hypothesis.extra.django import TestCase, from_model

from wishbone.data.models import DeviceTag

# Ideally, Django would provide us a way of testing the BaseTag model
# directly--or at least in a way that doesn't require a lot of manual,
# flaky configuration, but that has yet to be solved:
# https://code.djangoproject.com/ticket/7835
# So instead, we'll use the DeviceTag model -- hypothesis will still
# generate data for each derived BaseTag model for the fields they
# share, but ¯\_(ツ)_/¯


class TestBaseTag(TestCase):
    @given(from_model(DeviceTag))
    def test_base_tag(self, instance: DeviceTag) -> None:
        self.assertTrue(len(instance.name) > 0)
        print(type(instance.name))  # <class 'django.utils.functional.lazy.<locals>.__proxy__'> (then) <class 'django.utils.functional.lazy.<locals>.__proxy__'> (again)
        self.assertIsInstance(instance.name, str)  # <--- Fails here
        self.assertIsInstance(instance.enabled, bool)  # <--- But this if fine if the above line is commented

Models

class BaseTag(models.Model):
    """An abstract model containing attributes shared across all tags.

    Cannot be used directly and should be subclassed into a concrete
    model.
    """

    # Basic
    name = models.CharField(
        _("Name"),
        max_length=64,
        default=_("New Tag"),
        validators=(validators.MinLengthValidator(limit_value=1),),
    )
    enabled = models.BooleanField(_("Enabled"), default=True)
    # TODO: TagGroup

    class Meta(object):
        abstract = True


class DeviceTag(BaseTag):
    pass

[tybug]

Apologies for the delayed response here. Presumably you are using from django.utils.translation import gettext_lazy as _, in which case this is not a bug: the default value of DeviceTag is _("New Tag") which is indeed a functional lazy wrapper as you saw printed. You can force evaluation with str(instance.name) to get the non-lazy version.

[dakotahorstman]

Ah, I see.

Very good. I appreciate it. Interesting it happens on the BooleanField, but not the CharField. Perhaps the name parameter is treated differently by Django for some reason.

[tybug]

well, Hypothesis is using the default= parameter here to pick a value for the field, not the name parameter. default is _("New Tag") in name but True in enabled. That is why instance.name is a lazy wrapper while instance.enabled is a boolean.