Nautobot ORM actions slow when performed in the context of a Job after upgrading to v1.2

[wvandeun]

Environment

• Python version: 3.8.12
• Nautobot version: v1.2.5, v1.2.11

We are using Nautobot jobs to provision new sites, updating existing sites with new designs, etc etc.
After the upgrade to 1.2 (tested with 1.2.5 and 1.2.11), we noticed that these jobs take significantly more time to complete compared to 1.1 (tested with 1.1.6).

Specifically we see that certain ORM actions seem to be slower. We experience it when setting tagged VLANs on an interface or when setting a tag on an interface.Executing these actions outside of the context of a job (for example nbshell) behaves normally and fast.

vlans = VLAN.objects.filter(site__slug="somesite")[0:10]
interface = Interface.objects.get(
    name = "GigabitEthernet0/1",
    device = device
)

# slower
interface.tagged_vlans.set(vlans)

tags = {f"tag{i}" for i in range(10)}
for tag in tags:
   interface.tags.add(tag)

The slowness of the ORM actions in jobs seems to have a relation to the amount of change-log records in the db.
Attached you can find a job that we have used for testing.

• v1.1.6 (no change-log records) takes +-2m5s
• v1.1.6 (100k change-log records) takes +-2m
• v1.1.6 (3m change-log records) takes +-2m
• v1.2.5 (no change-log records) takes +- 2m30s
• v1.2.5 (100k change-log records) takes +- 4m
• v1.2.5 (3m change-log records) takes +- 40m (doesn't finish before NAUTOBOT_CELERY_TASK_SOFT_TIME_LIMIT is reached)

I did the tests on a completely dummy created data set, to exclude any weird things in our data set.

Steps to Reproduce

1. Create a significant amount of changelog records (>100K)
2. Execute this job in Nautobot v1.2 or above

Expected Behavior

Would expect the job to finish in the approximately the same time as when using earlier versions of Nautobot.

Observed Behavior

Job takes significantly more time to execute, compared to earlier versions of Nautobot.

[jathanism]

@wvandeun One quick question as I'm about to sit down with this: When you say changelog records you do mean specifically actual ObjectChange objects? I just wanted to clarify between those and JobLogEntry objects, which were introduced in v1.2, to store job log entries as individual rows.

I intend to review both, but just wanted that clarification. Thanks!

[wvandeun]

I mean ObjectChange objects with that.

[bryanculver]

Closed with #1641