Release: Merge release into master from: release/3.1.0#15162
Merged
Conversation
…-dev Release: Merge back 3.0.0 into dev from: master-into-dev/3.0.0-3.1.0-dev
* Added global required fields notice for WCAG H90 compliance * display_tags.py add and removed blank line * Added setting/env variable DD_SHOW_A11Y_REQUIRED_FIELDS_NOTICE --------- Co-authored-by: symon.vezina <symon.vezina@hrsdc-rhdcc.gc.ca>
…-dev Release: Merge back 3.0.1 into dev from: master-into-dev/3.0.1-3.1.0-dev
* perf(importers): batch vulnerability_id inserts Replace per-row Vulnerability_Id saves with bulk_create in two layers: - fix sanitize_vulnerability_ids to return filtered list (was a no-op bug — reassigned local variable, caller never saw the result) - save_vulnerability_ids now uses bulk_create per finding instead of one INSERT per ID; fixes all callers including the reimporter path - DefaultImporter.store_vulnerability_ids accumulates Vulnerability_Id objects across all findings in a batch; flush_vulnerability_ids() does a single bulk_create at each batch boundary (alongside location_handler.persist()) For a scan with 1000 findings × 5 CVEs each: 5000 INSERT queries reduced to O(batches) bulk_create calls. * perf(reimporter): batch vulnerability_id reconciliation Extend the cross-finding accumulation pattern to DefaultReImporter: - reconcile_vulnerability_ids now accumulates changed findings into pending_vuln_id_deletes / pending_vulnerability_ids instead of issuing per-finding DELETE + INSERT immediately - flush_vulnerability_ids (BaseImporter) runs one bulk DELETE WHERE finding_id IN (...) followed by one bulk_create for all new IDs - flush called at both dedupe batch boundaries (alongside location_handler.persist()) and after the mitigation loop Early-exit path (unchanged IDs) never touches either buffer, so the common case pays zero extra cost. Add two unit tests: cross-finding batch (3 findings, 2 changed + 1 unchanged, verify buffer contents before flush and DB state after) and unchanged-IDs early-exit (verify buffers stay empty). * test(performance): re-baseline importer query counts Remove pending-rebaseline skips from TestDojoImporterPerformanceSmall and TestDojoImporterPerformanceSmallLocations. Update all expected query counts to reflect the batch Vulnerability_Id insert optimisation (counts decrease by 1-20 queries per step depending on the scan size and code path). * fix(test): update TestSaveVulnerabilityIds mock for bulk_create The test mocked Vulnerability_Id.save (individual saves) but save_vulnerability_ids now uses bulk_create. Django's bulk_create validates FK references before issuing SQL, raising ValueError when the finding has no pk. Mock bulk_create instead and assert on the deduplicated object list passed to it. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…f tests (#14969) * perf(importers): batch BurpRawRequestResponse inserts + re-enable perf tests Replace per-finding save() calls in process_request_response_pairs with bulk_create at batch boundaries, mirroring the location_handler pattern. Reduces DB round-trips proportionally to findings with req/resp data. Drops the no-op clean() calls (BurpRawRequestResponse has no custom clean). Re-enable TestDojoImporterPerformanceSmall and TestDojoImporterPerformanceSmallLocations with recalibrated query counts after the RBAC→legacy authorization migration. * test(perf): recalibrate tag inheritance ZAP query counts Batch BurpRawRequestResponse inserts reduce per-finding saves for the ZAP parser (which emits req/resp pairs). Update expected counts to match.
…ocker-compose.yml) (#15022) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…(.github/workflows/release-x-manual-tag-as-latest.yml) (#14990) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
…v3.0.3 to v3.1.0 (.github/workflows/detect-merge-conflicts.yaml) (#14992) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
….23.1 (docker-compose.override.dev.yml) (#14997) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.15 to 0.15.16. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.15...0.15.16) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.16 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
Bumps [pdfmake](https://github.com/bpampuch/pdfmake) from 0.3.8 to 0.3.10. - [Release notes](https://github.com/bpampuch/pdfmake/releases) - [Changelog](https://github.com/bpampuch/pdfmake/blob/master/CHANGELOG.md) - [Commits](bpampuch/pdfmake@0.3.8...0.3.10) --- updated-dependencies: - dependency-name: pdfmake dependency-version: 0.3.10 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
Bumps [vcrpy](https://github.com/kevin1024/vcrpy) from 8.1.1 to 8.2.1. - [Release notes](https://github.com/kevin1024/vcrpy/releases) - [Changelog](https://github.com/kevin1024/vcrpy/blob/master/docs/changelog.rst) - [Commits](kevin1024/vcrpy@v8.1.1...v8.2.1) --- updated-dependencies: - dependency-name: vcrpy dependency-version: 8.2.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [sqlalchemy](https://github.com/sqlalchemy/sqlalchemy) from 2.0.50 to 2.0.51. - [Release notes](https://github.com/sqlalchemy/sqlalchemy/releases) - [Changelog](https://github.com/sqlalchemy/sqlalchemy/blob/main/CHANGES.rst) - [Commits](https://github.com/sqlalchemy/sqlalchemy/commits) --- updated-dependencies: - dependency-name: sqlalchemy dependency-version: 2.0.51 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…3 (.github/workflows/test-helm-chart.yml) (#15021) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Bumps [@tailwindcss/cli](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-cli) from 4.3.0 to 4.3.1. - [Release notes](https://github.com/tailwindlabs/tailwindcss/releases) - [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.3.1/packages/@tailwindcss-cli) --- updated-dependencies: - dependency-name: "@tailwindcss/cli" dependency-version: 4.3.1 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps vulners from 3.1.10 to 3.1.11. --- updated-dependencies: - dependency-name: vulners dependency-version: 3.1.11 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Move the Finding Groups links (Open/All/Closed Findings Groups) into the Findings dropdown in the new-UI sidebar and remove the standalone top-level "Finding Groups" nav item. A subtle divider separates the groups links within the dropdown. This declutters the top-level navigation and keeps related finding views together.
) Trivy legacy-format reports have no "Class" field, so the parser set unsaved_tags entries to None (e.g. ['debian', None]). clean_tags then crashed the whole import with TypeError in TAG_PATTERN.sub, after parsing had already succeeded (regression from #14111 in 2.55.0). - clean_tags: drop None entries instead of crashing (defends every parser) - trivy parser: filter falsy values at all four unsaved_tags sites - regression tests: clean_tags None handling; legacy fixture tags contain no None Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
….0-dev Release: Merge back 3.0.100 into dev from: master-into-dev/3.0.100-3.1.0-dev
…14999) Several new-UI templates hardcoded brand colors that should reference the design tokens defined in components/tailwind.css: - calendar.html: active engagement/test events used the legacy Bootstrap primary blue (#337ab7) instead of the brand blue. - benchmark.html: table link color, same legacy #337ab7. - base.html: sidebar background (#3864) and sub-nav link color (#82B0D9) hardcoded the Fuji Blue brand hues. These now use the matching var(--color-dd-primary-*) tokens (500/900/200). Token values are identical to the hardcoded hex, so there is no visual change. Intentionally left as-is: - The sidebar lighter link color (#C6DDF2): the matching shade (dd-primary-100) is not emitted in the compiled CSS, since Tailwind v4 only outputs theme variables referenced by a generated utility. - PDF report templates also contain #337ab7, but CSS custom properties do not resolve in the PDF renderer. - Generic white/black, neutral grays, and the custom #002a4d shade have no design token. FullCalendar 3.10.5 applies the event color verbatim as an inline style, so the variable resolves at render time. Verified in the running new UI: sidebar bg = rgb(0,56,100), sub-nav link = rgb(130,176,217), calendar event = rgb(23,121,197) -- each identical to the hex it replaced.
…gin page (#14998) The new Tailwind login page hardcoded hex color values that exactly duplicate the design tokens defined in components/tailwind.css. Swap them for the corresponding var(--color-*) tokens so the page stays in sync with the design system if the palette changes. The token values are identical to the previously hardcoded hex, so there is no visual change. The control-label color (#333333) and the alpha-channel rgba() shadows are left as-is since they have no exact token equivalent.
…v4.1.0 (.github/workflows/release-x-manual-tag-as-latest.yml) (#14991) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
Bumps [json-log-formatter](https://github.com/marselester/json-log-formatter) from 1.1.1 to 1.2.1. - [Release notes](https://github.com/marselester/json-log-formatter/releases) - [Commits](marselester/json-log-formatter@v1.1.1...v1.2.1) --- updated-dependencies: - dependency-name: json-log-formatter dependency-version: 1.2.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
…5027) Bumps [django-permissions-policy](https://github.com/adamchainz/django-permissions-policy) from 4.30.0 to 4.31.0. - [Changelog](https://github.com/adamchainz/django-permissions-policy/blob/main/CHANGELOG.rst) - [Commits](adamchainz/django-permissions-policy@4.30.0...4.31.0) --- updated-dependencies: - dependency-name: django-permissions-policy dependency-version: 4.31.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: rossops <20447042+rossops@users.noreply.github.com>
…ub/workflows/validate_docs_build.yml) (#15120) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…38.1 (helm/defectdojo/values.yaml) (#15119) * Update gcr.io/cloudsql-docker/gce-proxy Docker tag from 1.38.0 to v1.38.1 (helm/defectdojo/values.yaml) * update Helm documentation --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…15103) The global finding list ordered by sla_expiration_date seq-scanned and sorted the entire authorized finding set on every page load. Add a full (non-partial) btree on sla_expiration_date so the planner can walk the index and stop at the LIMIT instead. The existing partial idx_finding_sla_open_cov cannot serve this query because the query has no is_mitigated predicate. Measured on a 10M-finding clone, global finding list ORDER BY sla_expiration_date ASC LIMIT 25 (per-user authorization filter), under load test: - Before: mean 6591 ms, max 20732 ms, total 1582 s - After: mean 115 ms, max 4992 ms, total 96 s (~57x mean improvement; the query dropped from the #1 offender to negligible). Index size ~68 MB on the clone. Built CONCURRENTLY (atomic = False) so the migration takes no ACCESS EXCLUSIVE lock on the large dojo_finding table. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
…ithub/workflows/k8s-tests.yml) (#15121) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…5126) Bumps [django-permissions-policy](https://github.com/adamchainz/django-permissions-policy) from 4.31.0 to 4.32.0. - [Changelog](https://github.com/adamchainz/django-permissions-policy/blob/main/CHANGELOG.rst) - [Commits](adamchainz/django-permissions-policy@4.31.0...4.32.0) --- updated-dependencies: - dependency-name: django-permissions-policy dependency-version: 4.32.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [humanize](https://github.com/python-humanize/humanize) from 4.15.0 to 4.16.0. - [Release notes](https://github.com/python-humanize/humanize/releases) - [Commits](python-humanize/humanize@4.15.0...4.16.0) --- updated-dependencies: - dependency-name: humanize dependency-version: 4.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.19 to 0.15.20. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.19...0.15.20) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.20 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…15129) Bumps [tailwindcss](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss) from 4.3.1 to 4.3.2. - [Release notes](https://github.com/tailwindlabs/tailwindcss/releases) - [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.3.2/packages/tailwindcss) --- updated-dependencies: - dependency-name: tailwindcss dependency-version: 4.3.2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…/workflows/release-x-manual-docker-containers.yml) (#15138) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Bumps [@tailwindcss/cli](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-cli) from 4.3.1 to 4.3.2. - [Release notes](https://github.com/tailwindlabs/tailwindcss/releases) - [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.3.2/packages/@tailwindcss-cli) --- updated-dependencies: - dependency-name: "@tailwindcss/cli" dependency-version: 4.3.2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…15089) * feat(announcement): add DD_OS_MESSAGE_ENABLED and per-user dismiss for the OS promo banner DD_OS_MESSAGE_ENABLED (bool, default True) is a global admin opt-out: when False, get_os_banner() returns None before any network call, so no request is made to the GCS bucket. Default behavior is unchanged. Authenticated users can also dismiss the banner via a close (x) button. The dismissal is stored as a hash of the current message on UserContactInfo (mirroring the ui_use_tailwind preference), so the banner reappears only when the promo text changes. The button posts to a new dismiss_os_message endpoint (form-based CSRF) and hides the banner instantly via JS, degrading to a normal POST when JS is disabled. Includes migration 0270, docs and template-env updates, and unit tests (36 passing in unittests/test_os_message.py). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(announcement): left-align the OS banner dismiss as a bordered button Moves the dismiss (x) to lead the banner, before the headline, so it is clearly separated from the expand caret and avoids misclicks. Replaces the reused Bootstrap '.close' class (which collided with auto-generated v3 UI styles) with a dedicated 'os-message-dismiss' class styled as a small bordered button. Lays out the OS banner as a centered flex row scoped to [data-source="os"], so the styling applies only to this banner; other banners and the messages notifications are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(announcement): version the dismiss view test fixture for V3_FEATURE_LOCATIONS TestDismissOsMessageView loaded dojo_testdata.json directly, which raises NotImplementedError under V3_FEATURE_LOCATIONS=True (the fixture contains deprecated Endpoint records). Adding the @versioned_fixtures decorator swaps it to dojo_testdata_locations.json under V3, matching the convention used by the rest of the suite. Test-only change; verified passing locally under both V3=True and V3=False. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: store OS-banner dismissal in an extensible user_state_details JSON field Replace the dedicated UserContactInfo.os_message_dismissed_hash column with a single user_state_details JSONField that holds an open-ended set of small, per-user UI flags (dismissed banners, 'don't show again' toggles, ...). The banner-dismissal hash now lives under the OS_MESSAGE_DISMISSED_KEY key. This avoids adding a new column (and migration) for every minor per-user toggle; future ephemeral flags of this kind become a new JSON key with no schema change. Behavioral/queryable fields (block_execution, token_*, ...) intentionally stay as columns. * fix(api): exclude user_state_details JSONField from UserContactInfo filterset The new user_state_details JSONField broke UserContactInfoViewSet, which used filterset_fields = "__all__". django-filter's AutoFilterSet cannot build a filter for a JSONField and raised: AssertionError: AutoFilterSet resolved field 'user_state_details' with 'exact' lookup to an unrecognized field type JSONField. Replace the auto filterset with an explicit ApiUserContactInfoFilter that excludes the field (internal UI state, not meant to be filtered); all other fields keep their previous auto-filter behaviour. * fix: address review feedback on OS message opt-out - Renumber migration 0270 -> 0275 (depends on 0274_finding_sla_expiration_index) to resolve the leaf-node collision with the finding perf-index migrations that landed on dev. - Make the dismiss_os_message write concurrency-safe: read-modify-write user_state_details under an atomic() + select_for_update() row lock so a concurrent write to a different key can't clobber it (lost update). - Exclude the internal user_state_details JSON blob from UserContactInfoSerializer so it stays out of the public API. - Add a regression test asserting dismissal preserves other keys in user_state_details. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Valentijn Scholten <valentijnscholten@gmail.com> Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
…#15106) The new-UI filter categories used native <details>: the chevron animated but the body snapped open/closed, the header bars jumped to new positions instead of sliding together, and the categories packed inconsistently two-per-row (the filter form is display:flex/wrap), which was hard to scan. - Convert each category to a small Alpine toggle: a <button> with aria-expanded and type=button (so it never submits the filter form), with the body wrapped in an overflow-clipped grid that animates grid-template-rows 0fr->1fr. Animating the real layout height makes the whole header stack slide together smoothly and collapse fully to 0. Works in all browsers; default-open groups server-render the open state so there is no flash on load. - Give each category flex: 0 0 100% so they stack as full-width rows instead of packing two-per-row, making the filter panel easy to scan.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Matt Tesauro <mtesauro@gmail.com>
Collapse panels in the new UI (finding-detail sections and other
data-toggle="collapse" panels) toggled open/closed instantly. Add a smooth
height slide, consistent with the filter accordion.
- index.js: the shared collapse shim now animates the target height and
vertical padding between 0 and its natural size over 250ms, settling to auto
(open) or display:none via .in (closed). Respects prefers-reduced-motion
(instant toggle), preserves aria-expanded, guards against re-clicks
mid-animation, and handles both border-box and content-box. One change covers
every collapse panel in the new UI.
- base.html: cache-bust index.js with ?v={% static_v %} (matching how the CSS is
loaded) so the JS change reaches users; first-party JS was not versioned before.
No Bootstrap is reintroduced: this enhances the new UI vanilla-JS shim that
already replaced Bootstrap collapse.
…15136) The v3 Product->Asset relabel (#13155) centralises filter copy in dojo/asset/labels.py, gated by ENABLE_V3_ORGANIZATION_ASSET_RELABEL (default on). The Asset-level AND tag filter on the finding list (test__engagement__product__tags_and) was left with a hardcoded "Product Tags (AND)" label and help text, so it kept the legacy wording even with the relabel enabled -- inconsistent with its siblings "Test Tags (AND)" / "Engagement Tags (AND)" and with the OR variant, whose label is already set dynamically to "Tags (Asset)". Wire it to the edition-aware labels by adding ASSET_FILTERS_TAGS_ASSET_AND_LABEL and ASSET_FILTERS_TAGS_ASSET_AND_HELP to both vocabulary branches, so the field reads "Asset Tags (AND)" in v3 and remains "Product Tags (AND)" in the legacy edition. Adds a regression test asserting the rendered FindingFilter form field. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#15137) Since bootstrap.min.css is no longer loaded, the essential Bootstrap dropdown rules were re-implemented by hand in components/tailwind.css. That compat block omitted the `.dropdown-menu-right` utility (right: 0; left: auto), so every menu marked `dropdown-menu-right` (42 usages across 30+ templates, e.g. the finding actions menu) fell back to left-alignment and overflowed off the right edge of the page, clipping its labels and forcing horizontal page scroll. Restore the rule in the Bootstrap-compat block and rebuild tailwind-out.css (tailwindcss 4.2.4, matching the committed artifact). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ATURE_LOCATIONS (#15139) Creating a Finding Template from a finding or merging findings crashed with NotImplementedError when V3_FEATURE_LOCATIONS is enabled and the finding still carries legacy Endpoint rows (kept as backup by the locations migration). Both sites in dojo/finding/views.py iterated finding.endpoints.all() without the Endpoint.allow_endpoint_init() escape hatch used by the other legacy endpoint call sites. Fixes #15123. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…b/workflows/validate_docs_build.yml) (#15132) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…v7.1.0 to v7.2.0 (.github/workflows/release-3-master-into-dev.yml) (#15130) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…v1.35.6 (.github/workflows/k8s-tests.yml) (#14892) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* make impact_path deterministic * docs: add 3.1 upgrade note for deterministic JFrog Xray impact paths The JFrog Xray API Summary Artifact parser now sorts impact paths before selecting the first one, so file_path, description, and unique_id_from_tool (the deduplication key) stay stable across re-imports. Previously JFrog's arbitrary impact_path ordering could change the dedup key between scans and cause a single CVE to be re-imported as multiple separate findings. Document the behavior change and cleanup guidance in the 3.1 upgrade notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…5007) * feat(importers): add async-wait import execution mode Introduce a third import/reimport post-processing execution mode and make it configurable per request and via the user profile. - async (default): dispatch post-processing to the background, respond immediately - async_wait (new): dispatch to the background but wait for deduplication to finish before responding, so scan_added notifications and the returned statistics reflect the deduplicated state - sync: run post-processing inline in the web process Replace the legacy UserContactInfo.block_execution boolean with an import_execution_mode CharField; a data migration maps block_execution=True to the 'sync' mode. wants_block_execution() now derives from the sync mode, preserving global foreground-execution semantics for all async tasks. Add a request field import_execution_mode (ImportScanSerializer/ ReImportScanSerializer) resolved against the profile (request override wins), and a deduplication_complete boolean in the import/reimport response. The post_process_findings_batch task now stores its result (ignore_result=False) so async_wait can join on it via AsyncResult.get(); the join is bounded by the new DD_IMPORT_ASYNC_WAIT_TIMEOUT setting. * feat(importers): make scan_added notifications dedup-aware notify_scan_added used the importer's in-memory finding lists, whose duplicate flag is stale because deduplication runs on separately-fetched instances. Once deduplication has completed (sync or async_wait, signalled by deduplication_complete), refresh the duplicate flag from the database and split each action list into "real" and duplicate findings: - findings_new -> net-new (excludes deduplicated) - findings_new_duplicate / findings_reactivated_duplicate / findings_untouched_duplicate - finding_count -> recomputed to exclude new/reactivated duplicates (so an all-duplicate import sends scan_added_empty) In plain async mode (dedup not awaited) the lists are left untouched, preserving historical behavior. Mail and webhook scan_added templates render the new duplicate sections. Note: import/reimport response statistics are already dedup-accurate once awaited — Test.statistics / Test_Import.statistics query Finding and annotate the duplicate count directly; only the notification used stale in-memory data. * refactor(importers): await dedup before all import notifications Move wait_for_post_processing() ahead of the test_added notification dispatch (not just notify_scan_added) so both notifications sent during a fresh import are emitted after deduplication has completed in async_wait mode. The reimporter already orders the await before its single notification. * refactor: rename import_execution_mode to deduplication_execution_mode The mode governs whether the request waits for deduplication, so name it for that. Rename the UserContactInfo field, the request/serializer field, the ImporterOptions attribute and validator, the resolver, and the IMPORT_EXECUTION_MODE_* constants to DEDUPLICATION_EXECUTION_MODE_*. Add a RenameField migration (0271) rather than mutating the add/remove migrations introduced earlier on this branch. * refactor: rename DD_IMPORT_ASYNC_WAIT_TIMEOUT to DD_DEDUPLICATION_ASYNC_WAIT_TIMEOUT, default 60s * refactor: keep block_execution as global switch, deduplication_execution_mode for import dedup block_execution gates every async task (notifications, jira, grading, deletes, reindex, ...) via we_want_async, so it must remain. Restore it as the global "run all async tasks in the foreground" flag and make deduplication_execution_mode an independent field that only controls how import/reimport deduplication post-processing is dispatched and awaited. - Restore the block_execution field; wants_block_execution() reads it again. - resolve_deduplication_execution_mode() falls back to block_execution -> sync. - Replace the destructive remove migration (0270) with a seed-only data migration that maps block_execution=True -> deduplication_execution_mode 'sync'; keep both. - Restore fixtures, the profile form field, and the user detail view (both fields). - Revert the test sweep: tests that need global foreground use block_execution=True again; the dedicated deduplication_execution_mode tests are updated for the split. * docs: 2.60 upgrade notes for deduplication_execution_mode * fix(importers): honor profile deduplication_execution_mode for UI import/reimport The API resolves deduplication_execution_mode in the serializer, but the UI import/reimport views built their context straight from the form and never resolved it, so UI imports silently defaulted to async regardless of the user's profile. Resolve it from the profile in both UI process_form paths. * test: revert set_block_execution to the block_execution checkbox Option B keeps block_execution as a checkbox in the profile form, so the integration helper toggles id_block_execution again instead of the deduplication_execution_mode select (which no longer existed under that id). * refactor(migrations): split into schema add + data seed (2 migrations) Collapse the add/rename pair into a single AddField that creates deduplication_execution_mode with its final name, and keep the seed as a separate data migration, per the convention of keeping schema and data migrations apart. Drops the interim rename migration. * test: use versioned_fixtures so dedup mode tests pass under V3_FEATURE_LOCATIONS The CI 'true' matrix variant enables V3_FEATURE_LOCATIONS, where loading dojo_testdata.json (containing Endpoints) raises NotImplementedError. Decorate the test classes with @versioned_fixtures so the locations fixture is used in that mode, matching the other import/reimport test suites. * test(perf): add async_wait deduplication performance test Covers the 'async_wait' execution mode: post-processing is dispatched to a background worker and the request joins on it before responding. The dedup queries run in the worker (separate connection), not the web request, so the only web-side delta over the plain async path is the post-dedup notification refresh SELECT (+1): 109->110 first import, 89->90 second. Does not use CELERY_TASK_ALWAYS_EAGER (that would run the task inline on the request connection and wrongly count worker queries). The dedup batch is dispatched async and the join (AsyncResult.get) is mocked to return instantly, simulating a finished worker. Adds an optional dedup_mode param to _deduplication_performance. * fix(migrations): renumber import-execution-mode migrations onto dev (0270/0271) dev added 0269_normalize_blank_finding_components, colliding with this branch's 0269. Renumber to 0270_usercontactinfo_deduplication_execution_mode and 0271_seed_deduplication_execution_mode and repoint the dependency chain onto dev's 0269. * fix(importers): drop ignore_result=False override on post_process_findings_batch The async_wait deduplication join works with the global CELERY_TASK_IGNORE_RESULT=True default (verified live by reviewer against a real broker/worker). Removing the per-task override avoids writing a celery_taskmeta result row for every batch of every import (incl. plain async), which were retained for CELERY_RESULT_EXPIRES. * test(e2e): selenium integration test for async_wait deduplication mode Adds async_wait coverage to the existing tests/dedupe_test.py DedupeTest suite: set the user profile to 'async_wait', import a Checkmarx report into two tests of a dedupe-on-engagement engagement, and assert the duplicates are visible immediately (check_nb_duplicates_no_wait, no sleep/retry) — proving the request blocked until the cross-process deduplication finished. This exercises the real worker->request join that the eager unit tests cannot cover. Adds a set_deduplication_execution_mode() helper to base_test_class. * fix(importers): make async_wait actually join via per-dispatch ignore_result The async_wait join via AsyncResult.get() was a silent no-op: the global CELERY_TASK_IGNORE_RESULT=True means post_process_findings_batch never stores a result, so .get() returns immediately instead of blocking. The import responded with deduplication_complete=true while dedup had not run. Pass ignore_result=False only on the async_wait dispatch, threaded through dojo_dispatch_task to apply_async. The task decorator stays plain @app.task, so async/sync imports do not write useless celery_taskmeta result rows. * test(dedupe): replace Selenium async_wait test with real-worker API guard The Selenium async_wait test could not fail when the cross-process join was broken: with only 2 findings and several page navigations between the import and the duplicate check, the background dedupe finished regardless of whether the import actually blocked. It also never read deduplication_complete. Remove it and add ImportAsyncWaitApiTest: a pure-API test that imports a 400-finding report twice into a dedup-on-engagement engagement against the real worker, then asserts (no sleep/retry) that async_wait reports deduplication_complete=true with all 400 marked duplicate at response time, while plain async returns incomplete with fewer marked. A no-op join makes async_wait behave like async and fails the test. * test(dedupe): make async_wait integration test deterministic via gated dedup delay The giant-report approach was non-deterministic in CI: deduplication is dispatched per batch during the import, so on a fast worker plain `async` finishes dedup before the response and marks all findings too. The count discriminator then couldn't tell async_wait from async (the control failed `400 != 400`), and a broken async_wait could false-pass. Add DD_DEDUPLICATION_BATCH_PROCESS_TEST_DELAY (default 0, no-op in production), set to 3s on the integration-test celeryworker only. Each dedup batch sleeps before doing work, so async_wait blocks past it (all duplicates marked) while async returns mid-delay (none marked) — independent of worker speed. Rewrite ImportAsyncWaitApiTest to a small report and assert async marks exactly 0. Verified locally: green normally; fails `0 != 25` when the join fix is reverted. * test(dedupe): scope async_wait dedup delay to the test's findings, widen margin The first delay attempt broke two ways in CI: - It delayed *every* dedup batch in the UI suite, breaking the timing-sensitive close_old_findings_dedupe_test. - 3s was too short: the async control's import + observation GET took longer than that on CI, so dedup finished first and the control marked all findings. Add DD_DEDUPLICATION_BATCH_PROCESS_TEST_DELAY_FILTER (a finding-title prefix, case-insensitive) so only the async_wait test's own findings are delayed -- other dedupe tests run at full speed. Bump the delay to 10s for margin over the import round-trip. The Generic importer title-cases findings, so the match uses istartswith. Verified locally against the real worker: test_wait now blocks ~10s and marks all 25; async marks 0 at response time; reverting the join fix fails 0 != 25. * test(dedupe): log WARN when the test-only dedup delay fires Diagnostic for CI: surfaces exactly which post_process_findings_batch invocations hit the integration-test delay (and the finding count + filter), to confirm the async_wait test's batches are being delayed as intended. * test(dedupe): assert only deduplication_complete for the async control The async control's marked-count assertion was non-deterministic: its post_process_findings_batch task races the import request's own DB commit, so on CI the findings were not visible when the delay filter ran (no sleep) yet were deduplicated moments later -> the control saw all 25 marked and failed. Drop the count assertion for async and keep only deduplication_complete is False, which is deterministic and mode-based. The duplicate-count guarantee stays on the async_wait test, where the worker-side delay (confirmed firing via the WARN log: 10s/25 findings per batch) makes it robust and a no-op join still fails 0 != NUM_FINDINGS. * test(dedupe): note async_wait eager test is not a real guard Document that test_import_async_wait_returns_statistics cannot fail when the cross-process join is broken (eager .get() always returns inline), pointing to the real-worker ImportAsyncWaitApiTest. Kept for endpoint/field plumbing coverage and future reference. * docs(dedupe): document import/reimport deduplication execution mode Extend the deduplication 'Background processing' section with the new deduplication_execution_mode (async / async_wait / sync), the per-profile and per-request override, the deduplication_complete response field, the DD_DEDUPLICATION_ASYNC_WAIT_TIMEOUT bound, and its relationship to the global block_execution flag. Previously this was only in the 2.60 upgrade note. * feat(importers): log async_wait deduplication wait duration Record how long the import request blocked on the deduplication join: - DEBUG line with elapsed seconds + task count + success on completion. - the timeout/error WARNING now includes the elapsed time before it gave up. Timing detail is DEBUG so it does not add noise per import in production; the timeout case stays at WARNING since it signals a degraded (respond-anyway) join. * chore: stop tracking local-only files committed by mistake CLAUDE.local.md, fix-jira-push-eligibility.md, phase10-session-prompt.md, scripts/run-nuclei-docker.sh and tagoulus-replcement-options.md were accidentally added in an earlier refactor commit; none belong in this feature. Remove them from version control (kept on disk locally) and gitignore CLAUDE.local.md / CLAUDE.md so they cannot be re-added. * fix(migrations): renumber dedup-execution-mode migrations onto current dev leaf dev advanced past the previous 0270/0271 numbering (it now ships 0270_finding_visibility_perf_indexes through 0272_reencrypt_tool_config_credentials_aes_gcm), so the PR's 0270/0271 migrations reintroduced a second leaf node off 0269_normalize_blank_finding_components -> 'Conflicting migrations detected; multiple leaf nodes in the migration graph'. Renumber onto the current leaf: - 0270_usercontactinfo_deduplication_execution_mode -> 0273_..., dep 0272_reencrypt_tool_config_credentials_aes_gcm - 0271_seed_deduplication_execution_mode -> 0274_..., dep 0273_usercontactinfo_deduplication_execution_mode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(perf): bump async_wait dedup query baseline to match measured counts CI 'Check counts are up to date' measured 94/74 for test_deduplication_performance_pghistory_async_wait (first/second import) vs the committed 93/73. The web-side delta over plain async (92/72) is +2, not +1: the post-dedup notification refresh SELECT plus one result-backend write from the per-dispatch ignore_result=False that enables the AsyncResult.get() join. Update the baseline and the docstring accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(migrations): renumber dedup-execution-mode migrations onto current dev leaf Rebasing onto dev pulled in 0273_product_upper_name_index, 0274_finding_sla_expiration_index and 0275_usercontactinfo_user_state_details, colliding with this branch's 0273/0274 and producing two leaf nodes. Renumber onto the current leaf: - 0273_usercontactinfo_deduplication_execution_mode -> 0276_..., dep 0275_usercontactinfo_user_state_details - 0274_seed_deduplication_execution_mode -> 0277_..., dep 0276_usercontactinfo_deduplication_execution_mode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move deduplication_execution_mode upgrade note to 3.1 The feature now ships in 3.1.0, not 2.60. Revert 2.60.md to its "no special instructions" state and fold the deduplication execution mode section into 3.1.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve conflicts from the finding-views reorg: - dojo/finding/views.py is now a re-export shim on dev; kept the shim. - Ported the mktemplate/merge endpoint guards from #15139 into their new home dojo/finding/ui/views.py (Endpoint.allow_endpoint_init()), since the bug fix landed in the old file location. - helm/defectdojo/Chart.yaml: kept dev's populated artifacthub.io/changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Release 3.1.0: resolved bugfix into dev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release triggered by
rossops