bug with django 4.1

[hishamkaram]

Description

we are using factory boy with django unit test but when we upgraded from 4.0.4 to 4.1.1
i got the following errors

Traceback (most recent call last):
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 928, in get_or_create
    return self.get(**kwargs), False
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 650, in get
    raise self.model.DoesNotExist(
src.core.timezone.models.TimeZone.DoesNotExist: TimeZone matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "core_timezone_pkey"
DETAIL:  Key (id)=(4) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/django.py", line 143, in _get_or_create
    instance, _created = manager.get_or_create(*args, **key_fields)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 935, in get_or_create
    return self.create(**params), True
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 671, in create
    obj.save(force_insert=True, using=self.db)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 831, in save
    self.save_base(
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 882, in save_base
    updated = self._save_table(
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 1025, in _save_table
    results = self._do_insert(
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 1066, in _do_insert
    return manager._insert(
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 1790, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1657, in execute_sql
    cursor.execute(sql, params)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 103, in execute
    return super().execute(sql, params)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/cacheops/transaction.py", line 98, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "core_timezone_pkey"
DETAIL:  Key (id)=(4) already exists.

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/test/testcases.py", line 1448, in setUpClass
    cls.setUpTestData()
  File "/opt/atlassian/pipelines/agent/build/adpp_backend/src/tests.py", line 47, in setUpTestData
    cls._admin_user = cls._create_user()
  File "/opt/atlassian/pipelines/agent/build/adpp_backend/src/tests.py", line 41, in _create_user
    return AdminUserFactory()
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/base.py", line 40, in __call__
    return cls.create(**kwargs)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/base.py", line 528, in create
    return cls._generate(enums.CREATE_STRATEGY, kwargs)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/django.py", line 120, in _generate
    return super()._generate(strategy, params)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/base.py", line 465, in _generate
    return step.build()
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 260, in build
    step.resolve(pre)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 201, in resolve
    self.attributes[field_name] = getattr(self.stub, field_name)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 346, in __getattr__
    value = value.evaluate_pre(
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 48, in evaluate_pre
    return self.evaluate(instance, step, context)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 411, in evaluate
    return step.recurse(subfactory, extra, force_sequence=force_sequence)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 218, in recurse
    return builder.build(parent_step=self, force_sequence=force_sequence)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 260, in build
    step.resolve(pre)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 201, in resolve
    self.attributes[field_name] = getattr(self.stub, field_name)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 346, in __getattr__
    value = value.evaluate_pre(
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 48, in evaluate_pre
    return self.evaluate(instance, step, context)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 411, in evaluate
    return step.recurse(subfactory, extra, force_sequence=force_sequence)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 218, in recurse
    return builder.build(parent_step=self, force_sequence=force_sequence)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 260, in build
    step.resolve(pre)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 201, in resolve
    self.attributes[field_name] = getattr(self.stub, field_name)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 346, in __getattr__
    value = value.evaluate_pre(
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 48, in evaluate_pre
    return self.evaluate(instance, step, context)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 411, in evaluate
    return step.recurse(subfactory, extra, force_sequence=force_sequence)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 218, in recurse
    return builder.build(parent_step=self, force_sequence=force_sequence)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 264, in build
    instance = self.factory_meta.instantiate(
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/base.py", line 317, in instantiate
    return self.factory._create(model, *args, **kwargs)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/django.py", line 166, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/django.py", line 147, in _get_or_create
    for lookup, value in cls._original_params.items()
AttributeError: type object 'TimeZoneFactory' has no attribute '_original_params'

To Reproduce

Share how the bug happened:

Model / Factory code

# Include your factories and models here
import factory
from factory.faker import faker

from .models import TimeZone

fake = faker.Faker()


class TimeZoneFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = TimeZone

    name = factory.Sequence(lambda n: f'{fake.name()}_{n}')

#models

class TimeZone(BaseModel):
    """
    TimeZone
    """

    class Meta:
        db_table = 'core_timezone'
        verbose_name = _('time zone')
        verbose_name_plural = _('time zones')

    name = models.TextField(_('name'), unique=True)

The issue

this factory is used everywhere in the project, not sure why this error appeared after upgrading to django 4.1.1

[cunla]

The package introduced support for django 4.1 but it was not published yet.
Are there plans to publish a new version?

[francoisfreitag]

See #914.

[francoisfreitag]

The step to reproduce lack details. For example, we see in the stack trace that get_or_create is used, but the factories from the STR do not use that feature. Please provide enough details to reproduce the issue. Ideally, a test case, or at least a minimal project where the issue can be reproduced.

Shot in the dark: did you override the _generate() hook in your factory and not call super()._generate()? The cls._original_params are defined there:

factory_boy/factory/django.py

Lines 115 to 120 in 8e5b79a

	@classmethod
	def _generate(cls, strategy, params):
	# Original params are used in _get_or_create if it cannot build an
	# object initially due to an IntegrityError being raised
	cls._original_params = params
	return super()._generate(strategy, params)

[hishamkaram]

@francoisfreitag Hi,
this Factory is a part of large code base, everything was working fine before upgrading to Django 4.1. I installed Django4.1.1 and run Django unit test as usual, i got those errors in the issue description.
NOTE: we didn’t override _generate()
i will try to provide a test case.

[francoisfreitag]

May be the same issue as #979.

[hmpf]

I get the same issue when going from Django 3.2.16 to 4.0.8, with factory_boy 3.2.1.

[hmpf]

I have found the reason why I see this.

Django 3.2 allows accessing foreign keys before an object is saved for the first time. Django 4.0 does not. I worked around this by saving twice in save(), and that triggers the IntegrityError.

Example:

class Directory(models.Model):
    name = models.CharField(max_length=255)
    parent = models.ForeignKey('self', null=True)
    num_subdirectories = models.IntegerField(default=0)
 
    # works on 3.2
    def save(self, *args, **kwargs):
        self.num_subdirectories = self.directory_set.count()
        super().save(*args, **kwargs)

    # needed for 4.0
    def save(self, *args, **kwargs):
        if not self.id:
            super().save(*args, **kwargs)
        self.num_subdirectories = self.directory_set.count()
        super().save(*args, **kwargs)

[foarsitter]

@hishamkaram @hmpf is there any chance your problem is solved with #981

Perhaps you can run you code against that branch?

[hishamkaram]

@francoisfreitag _original_params errors disappeared but still getting this UniqueViolation error. I don’t pass any id, I am just using create_batch and SubFactory in several test cases

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "core_timezone_pkey"
DETAIL:  Key (id)=(1) already exists.

[francoisfreitag]

It looks like an issue with your factory. get_or_create lookup fails, tries to insert a new object that violates the unique constraint.
If you think this is a bug from factory boy, please provide a minimal project / script to reproduce the example, or (even better) a failing test in the test suite.

[hishamkaram]

@francoisfreitag actually there is no get_or_create lookup in the factory but i will try to check how can i reproduce

[foarsitter]

Does your database have an autoincrement for TimeZone.id or is your sequence not in sync? It complains about that id=1 already exists and as far as I know Django does not generate primary keys but the database does.

[hishamkaram]

@francoisfreitag @foarsitter true, thank you so much. i think this branch fixed the issue

[foarsitter]

Thanks for testing and your feedback @hishamkaram!