Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Product Metrics: Performance Enhancements #10059

Merged
merged 4 commits into from
Apr 29, 2024

Conversation

blakeaowens
Copy link
Contributor

@blakeaowens blakeaowens commented Apr 29, 2024

[sc-5692]

This PR improves performance for Product metrics when there is a large number of Findings under a given DefectDojo Product. The following changes were made to improve performance:

1. Limiting Queries & Flattening of QuerySet(s)

Previously, the Product Metrics view relied on calls to the finding_querys utility. All of these queries have been instantiated at the beginning of the view as flattened lists, containing only the Finding field values required for metrics calculations. These original queries are then used throughout the Product metrics view (rather than relying on the finding_querys utility to fetch Findings over and over).

2. Removal of in Operator

The in operator in Python is time complexity O(n) on average (and called 3 times within the for-loop across all Findings). To eliminate this, three dictionaries in the format of { finding_id: True } were constructed to eliminate the need to call Python's in operator. Instead, finding in open_findings can be replaced with if open_findings.get(finding_id, None), which drastically improves performance.

3. Rework of open_objs_by_age Construction

The open_objs_by_age dataset, used to display Finding counts by age, was previously time complexity O(n^3). This performance bottleneck has been eliminated entirely by moving the construction of open_objs_by_age into an existing O(n) for-loop.

4. Removal of |length from Template

Previously, entire query sets were passed to the product_metrics.html template, then used to display total Finding counts in the headers/footers of plots using the |length display tag. These counts are now calculated in the view by taking the len() of the newly flattened lists mentioned in point #1.

Other minor organizational changes and cleanup made in addition to the 4 points above.

@github-actions github-actions bot added the ui label Apr 29, 2024
@blakeaowens blakeaowens changed the title product metrics improvements Product Metrics: Performance Enhancements Apr 29, 2024
Copy link

Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.

DryRun Security Status Findings
Configured Codepaths Analyzer 0 findings
Sensitive Files Analyzer 0 findings
AppSec Analyzer 0 findings
Authn/Authz Analyzer 2 findings
Secrets Analyzer 0 findings

Note

🟢 Risk threshold not exceeded.

Powered by DryRun Security

Copy link
Contributor

@mtesauro mtesauro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@Maffooch Maffooch merged commit c76a68a into DefectDojo:bugfix Apr 29, 2024
122 checks passed
mwager added a commit to mwager/django-DefectDojo that referenced this pull request Apr 30, 2024
… kiuwan-sca

* 'kiuwan-sca' of github.com:mwager/django-DefectDojo:
  Update versions in application files
  Product Metrics: Performance Enhancements (DefectDojo#10059)
  String Based Filtering: Follow on for DefectDojo#10038 (DefectDojo#10050)
  update semgrep tests (DefectDojo#10058)
  Jira Webhook: Reorg logging and responses (DefectDojo#10049)
  Similar Findings: Create Toggle (DefectDojo#10047)
  Bump social-auth-app-django from 5.4.0 to 5.4.1 (DefectDojo#10026)
  Update versions in application files
  Update versions in application files
  Updated DryRun Security config (DefectDojo#10037)
  Filtering Performance: Add opt-in setting for converting to string ba… (DefectDojo#10038)
  Updates to semgrep parser (DefectDojo#10033)
  Update versions in application files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants