Optimize bulk updates of custom field values when custom fields are added/removed

[jeremystretch]

NetBox version

v4.2.5

Feature type

Change to existing functionality

Proposed functionality

This FR intends to capture an opportunity to improve performance when a custom field is added to or removed from a model which has many thousands of records. Whenever a custom field is added, its populate_initial_data() method is called to automatically populate the field's default value on all objects:

netbox/netbox/extras/models/customfields.py

Lines 279 to 289 in 7db0765

	def populate_initial_data(self, content_types):
	"""
	Populate initial custom field data upon either a) the creation of a new CustomField, or
	b) the assignment of an existing CustomField to new object types.
	"""
	for ct in content_types:
	model = ct.model_class()
	instances = model.objects.exclude(**{'custom_field_data__contains': self.name})
	for instance in instances:
	instance.custom_field_data[self.name] = self.default
	model.objects.bulk_update(instances, ['custom_field_data'], batch_size=100)

The current approach is suboptimal, as it requires loading each individual object into memory to effect the change. I believe we can optimize this by mutating the JSON data directly within the query:

from django.db.models import F, Func, Value
from django.db.models import JSONField

def bulk_update_cf_value(model, field_name, new_value):
    model.objects.update(
        custom_field_data=Func(
            F('custom_field_data'),
            Value([field_name]),
            Value(new_value, JSONField()),
            function='jsonb_set'
        )
    )

bulk_update_cf_value(Site, "test1", "abc123")

This allows us to update the value of a specific key for all objects without first loading each into memory.

Similar approaches will be needed for the remove_stale_data() and rename_object_data() methods on CustomField as well.

Use case

This should result in a dramatic reduction in the time required to bulk update custom field data, particularly when many thousands of records are affected.

Database changes

N/A

External dependencies

N/A