-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Product Metrics: Performance Enhancements #10059
Merged
Maffooch
merged 4 commits into
DefectDojo:bugfix
from
blakeaowens:product-metrics-improvements
Apr 29, 2024
Merged
Product Metrics: Performance Enhancements #10059
Maffooch
merged 4 commits into
DefectDojo:bugfix
from
blakeaowens:product-metrics-improvements
Apr 29, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
blakeaowens
changed the title
product metrics improvements
Product Metrics: Performance Enhancements
Apr 29, 2024
Hi there 👋, @DryRunSecurity here, below is a summary of our analysis and findings.
Note 🟢 Risk threshold not exceeded. Powered by DryRun Security |
grendel513
approved these changes
Apr 29, 2024
mtesauro
approved these changes
Apr 29, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved
Maffooch
approved these changes
Apr 29, 2024
devGregA
approved these changes
Apr 29, 2024
cneill
approved these changes
Apr 29, 2024
mwager
added a commit
to mwager/django-DefectDojo
that referenced
this pull request
Apr 30, 2024
… kiuwan-sca * 'kiuwan-sca' of github.com:mwager/django-DefectDojo: Update versions in application files Product Metrics: Performance Enhancements (DefectDojo#10059) String Based Filtering: Follow on for DefectDojo#10038 (DefectDojo#10050) update semgrep tests (DefectDojo#10058) Jira Webhook: Reorg logging and responses (DefectDojo#10049) Similar Findings: Create Toggle (DefectDojo#10047) Bump social-auth-app-django from 5.4.0 to 5.4.1 (DefectDojo#10026) Update versions in application files Update versions in application files Updated DryRun Security config (DefectDojo#10037) Filtering Performance: Add opt-in setting for converting to string ba… (DefectDojo#10038) Updates to semgrep parser (DefectDojo#10033) Update versions in application files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[sc-5692]
This PR improves performance for Product metrics when there is a large number of Findings under a given DefectDojo Product. The following changes were made to improve performance:
1. Limiting Queries & Flattening of QuerySet(s)
Previously, the Product Metrics view relied on calls to the
finding_querys
utility. All of these queries have been instantiated at the beginning of the view as flattened lists, containing only the Finding field values required for metrics calculations. These original queries are then used throughout the Product metrics view (rather than relying on thefinding_querys
utility to fetch Findings over and over).2. Removal of
in
OperatorThe
in
operator in Python is time complexityO(n)
on average (and called 3 times within the for-loop across all Findings). To eliminate this, three dictionaries in the format of{ finding_id: True }
were constructed to eliminate the need to call Python'sin
operator. Instead,finding in open_findings
can be replaced withif open_findings.get(finding_id, None)
, which drastically improves performance.3. Rework of
open_objs_by_age
ConstructionThe
open_objs_by_age
dataset, used to display Finding counts by age, was previously time complexityO(n^3)
. This performance bottleneck has been eliminated entirely by moving the construction ofopen_objs_by_age
into an existingO(n)
for-loop.4. Removal of
|length
from TemplatePreviously, entire query sets were passed to the
product_metrics.html
template, then used to display total Finding counts in the headers/footers of plots using the|length
display tag. These counts are now calculated in the view by taking thelen()
of the newly flattened lists mentioned in point#1
.Other minor organizational changes and cleanup made in addition to the 4 points above.