Skip to content

perf(tags): bulk-propagate inherited tags + gate child post_save on create#14812

Open
valentijnscholten wants to merge 1 commit intoDefectDojo:devfrom
valentijnscholten:perf/tag-inheritance-phase-a-bulk
Open

perf(tags): bulk-propagate inherited tags + gate child post_save on create#14812
valentijnscholten wants to merge 1 commit intoDefectDojo:devfrom
valentijnscholten:perf/tag-inheritance-phase-a-bulk

Conversation

@valentijnscholten
Copy link
Copy Markdown
Member

@valentijnscholten valentijnscholten commented May 5, 2026

This is going to be multi stage optimization to keep the PRs a little smaller to ease review.

This PR basically adds another batch tag method to apply inherited tags more efficiently.

Summary

  • Replaces the per-row obj.save() loop in dojo/product/helpers.py:propagate_tags_on_product_sync with bulk SQL through the existing tag-utils helpers. For every child model (Engagement / Test / Finding / Endpoint / Location), reads current inherited_tags in one query, computes the per-child diff against the Product's tags, and applies adds/removes via bulk_add_tag_mapping and the new bulk_remove_tags_from_instances helper. Both tags and inherited_tags fields are kept in sync.
  • Adds bulk_remove_tags_from_instances to dojo/tag_utils.py, symmetric to bulk_add_tags_to_instances. Decrements Tagulous tag counts to avoid drift, deletes through-model rows in one DELETE per tag-batch, invalidates prefetch caches.
  • Gates the per-child inherit_tags_on_instance post_save handler on created=True. The previous behavior fired on every save (create OR update), repeatedly re-applying inherited tags to children whose tag state had not changed. Sticky enforcement on user-driven tag edits is unchanged (still handled by make_inherited_tags_sticky on m2m_changed).
  • Lowers two pinned query-count baselines in unittests/test_tag_inheritance_perf.py to reflect the reductions.

Query-count impact

Hot path Before After Reduction
Product tag add -> 100 findings 4758 91 ~52x
Product tag remove -> 100 findings 4540 53 ~85x
Create 1 finding under inheritance 64 64 (Phase B target)
Create 100 findings under inheritance 4025 4025 (Phase B target)
Finding add user tag (sticky) 17 17 (Phase B target)
Finding remove inherited tag (sticky re-add) 44 44 (Phase B target)

Sticky and child-creation paths are unchanged in this PR. Phase B targets those (centralized inheritance module + drop the duplicate inherited_tags TagField).

Verification

  • unittests.test_tag_inheritance - 36 tests pass (5 skipped for v3 feature flag)
  • unittests.test_tag_utils_bulk - 29 tests pass
  • unittests.test_tag_inheritance_perf - 6 tests pass with the lowered pins

Behavior preserved

  • Inheritance flags (system + per-product) gate propagation as before.
  • inherited_tags field semantics unchanged - still a duplicate TagField kept in lock-step with tags. Phase B replaces it with a JSON column.
  • Sticky enforcement (re-add inherited tag if a user removes it from a child) still runs through the existing make_inherited_tags_sticky m2m_changed handler.
  • Locations with multiple linked products still receive the union of all related products' tags.

…reate

Replaces the per-row `.save()` loop in `propagate_tags_on_product_sync`
with bulk SQL through the existing tag-utils helpers. For every child
model (Engagement/Test/Finding/Endpoint/Location), reads current
inherited_tags in one query, computes the per-child diff against the
Product's tags, and applies adds/removes via `bulk_add_tag_mapping` and
the new `bulk_remove_tags_from_instances` helper. Both `tags` and
`inherited_tags` fields are kept in sync.

Also gates the per-child `inherit_tags_on_instance` post_save handler on
`created=True`. The previous behavior fired on every save (create OR
update), repeatedly re-applying inherited tags to children whose tag
state had not changed. Sticky enforcement on user-driven tag edits is
unchanged (still handled by `make_inherited_tags_sticky` on m2m_changed).

Pinned query-count baselines from PR DefectDojo#14811 drop accordingly:

  product_tag_add  -> 100 findings : 4758 -> 91   (~52x fewer queries)
  product_tag_remove -> 100 findings : 4540 -> 53 (~85x fewer queries)

Sticky and child-creation paths are unchanged in this PR. Phase B
targets those (centralized inheritance module + drop the duplicate
`inherited_tags` TagField).
@valentijnscholten valentijnscholten force-pushed the perf/tag-inheritance-phase-a-bulk branch from d2c4fc5 to bd62203 Compare May 7, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants