AWS Security Hub parser add unique_id_from_tool and deduplication #4376

nlandais · 2021-04-27T21:45:22Z

Add name of the AWS resource in the title for ease of use and to help in the remediation process
Dedupe finding on unique_id_from_tool: use the SecHub finding id (value after the last / from the finding's ARN)

…ctDojo in debug mode

…into dev

…ing DefectDojo in debug mode" This reverts commit 081fefe.

…into dev

- Add name of the AWS resource in the title for ease of use and to help in the remediation process - Dedupe on unique_id_from_tool: use the SecHub finding id (value after the last / from the finding's ARN) - Remove logiv aimed at closing the findings because it does actually work and competes for the close_old_findings logic

dojo/urls.py

dojo/tools/awssecurityhub/parser.py

valentijnscholten

comments

dojo/tools/awssecurityhub/parser.py

…into dev

damiencarol

Could you add some checks in the unit tests that show the new data in the findings?

.gitmodules

…into dev

nlandais · 2021-05-18T14:52:25Z

Could you add some checks in the unit tests that show the new data in the findings?
@damiencarol Bonjour Damien, pourrais-tu elaborer sur la requete consistant a modifier les unittests pour le filtre d'ingestion des donnees provenant de AWS SecurityHub ?
The parser does not change much in functionality, the big change in this PR is the fact that the deduplication and filtering was not working properly when ingesting one report at a time from AWS SecurityHub.
The behavior observed prior to this change request was as follows:

When drilling down into to a finding , the "imilar findings" section showed all the other AWS Security Hub findings regardless of the AWS resource associated with the finding.
Each new imported finding marked the previous one as duplicate, which is consistent with the behavior reported in the first bullet point.
In conclusion, there isn't any new data in the findings, the proposed change implements the behavior we come to expect with DefectDojo when it comes to aggregating similar findings, managing duplicate findings and closing stale/old findings.
Feel free to DM me on the OWASP defectodjo Slack channel (handle: Nicolas Landais) to discuss further.
-Merci

damiencarol

As you are adding modifications to unique_id_from_tool attribute, could you add some checks in the unit tests that checks these new data?

dojo/tools/awssecurityhub/parser.py

damiencarol · 2021-05-22T12:10:39Z

@nlandais there is something out of scope in the PR. it seems that one sub-module for documentation is updated.

…e_id_from_tool in the DB and used by the deduplication algorithm

Implementing review comment, by keeping complete ARN for the finding_id Making the resourc_id value a bit easier to read by splitting the ARN

dojo/tools/awssecurityhub/parser.py

Re-instating the mitigated=mitigated, in the finding properties

Add ing boolean to finding property to indicate the mitigation status (when mitigation date is not NULL, mitigation status gets set to TRUE)

damiencarol · 2021-06-16T15:09:13Z

@valentijnscholten could you take another look, @nlandais made a lot of modifications since the first reviews.

github-actions · 2021-06-17T20:55:22Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2021-06-18T13:23:33Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

Nicolas Landais and others added 6 commits February 1, 2021 14:19

Adding query_toolbar to expose the database queries when running Defe…

081fefe

…ctDojo in debug mode

Merge branch 'dev' of https://github.com/DefectDojo/django-DefectDojo …

36b0d1f

…into dev

Revert "Adding query_toolbar to expose the database queries when runn…

52a1045

…ing DefectDojo in debug mode" This reverts commit 081fefe.

Merge branch 'dev' of https://github.com/DefectDojo/django-DefectDojo …

6596314

…into dev

Removing unused module

1603a29

valentijnscholten reviewed Apr 28, 2021

View reviewed changes

dojo/urls.py Outdated Show resolved Hide resolved

valentijnscholten reviewed Apr 28, 2021

View reviewed changes

dojo/tools/awssecurityhub/parser.py Outdated Show resolved Hide resolved

valentijnscholten suggested changes Apr 28, 2021

View reviewed changes

damiencarol requested changes Apr 28, 2021

View reviewed changes

dojo/tools/awssecurityhub/parser.py Outdated Show resolved Hide resolved

dojo/tools/awssecurityhub/parser.py Outdated Show resolved Hide resolved

Nicolas Landais added 3 commits April 28, 2021 09:12

Merge branch 'dev' of https://github.com/DefectDojo/django-DefectDojo …

2447ded

…into dev

PR 4376, re-insterting line deleted by mistake

570129f

PR 4376, removing impact affectation and setting new findings as active

f423775

madchap closed this May 3, 2021

madchap reopened this May 3, 2021

Nicolas Landais added 2 commits May 3, 2021 11:26

Reverting change

fb64a8a

Fixing PEP8 finding

23832b6

nlandais requested review from damiencarol and valentijnscholten May 3, 2021 17:27

Nicolas Landais added 3 commits May 3, 2021 15:56

Merge branch 'dev' of https://github.com/DefectDojo/django-DefectDojo …

37820af

…into dev

Using HTTPS for Git instead of SSH

79f3129

Reworking submodules

46e0fbc

damiencarol requested changes May 4, 2021

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

nlandais requested a review from damiencarol May 4, 2021 17:51

Nicolas Landais added 2 commits May 18, 2021 10:17

Merge branch 'dev' of https://github.com/DefectDojo/django-DefectDojo …

d9e0c3f

…into dev

Reverting the gitmodule url https -> git

1c65f43

nlandais changed the title ~~Simplifying the parser for AWS Security Hub findings:~~ AWS Security Hub parser setting changes May 20, 2021

Sorry Docsy, did not mean to nuke you out of the tree

877e68b

damiencarol changed the title ~~AWS Security Hub parser setting changes~~ AWS Security Hub parser add unique_id_from_tool and deduplication May 22, 2021

damiencarol added the enhancement label May 22, 2021

damiencarol requested changes May 22, 2021

View reviewed changes

dojo/tools/awssecurityhub/parser.py Outdated Show resolved Hide resolved

Nicolas Landais and others added 8 commits May 26, 2021 14:34

Back to 1755d44f684e917457523d4bbd6e330754cefd8e of the Docsy submodule

329a918

Use the full ARN as the resource_id which will end up being the uniqu…

59388c7

…e_id_from_tool in the DB and used by the deduplication algorithm

Using the complete ARN as finding_id

ce77da5

Implementing review comment, by keeping complete ARN for the finding_id Making the resourc_id value a bit easier to read by splitting the ARN

add unit test

87775a0

Changing assert value and PEP8/linting update

49e7989

fix unit test

6417b04

Update test_awssecurityhub.py

dd9548e

Full ARN

c6d7e0a

nlandais requested a review from damiencarol June 2, 2021 02:14

damiencarol requested changes Jun 16, 2021

View reviewed changes

dojo/tools/awssecurityhub/parser.py Outdated Show resolved Hide resolved

nlandais added 3 commits June 16, 2021 09:52

Mitaged state had been deleted by error

4f120d4

Re-instating the mitigated=mitigated, in the finding properties

Adding mitigation status

17aca1b

Add ing boolean to finding property to indicate the mitigation status (when mitigation date is not NULL, mitigation status gets set to TRUE)

Update parser.py

00d98b8

damiencarol approved these changes Jun 16, 2021

View reviewed changes

nlandais requested a review from damiencarol June 16, 2021 14:19

Correcting property's name capitalization

c6fdab7

github-actions bot added the conflicts-detected label Jun 17, 2021

Merge branch 'dev' into dev

553a1c6

github-actions bot removed the conflicts-detected label Jun 18, 2021

valentijnscholten approved these changes Jun 18, 2021

View reviewed changes

valentijnscholten merged commit de9eb94 into DefectDojo:dev Jun 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS Security Hub parser add unique_id_from_tool and deduplication #4376

AWS Security Hub parser add unique_id_from_tool and deduplication #4376

nlandais commented Apr 27, 2021 •

edited

Loading

valentijnscholten left a comment

damiencarol left a comment

nlandais commented May 18, 2021

damiencarol left a comment

damiencarol commented May 22, 2021

damiencarol commented Jun 16, 2021

github-actions bot commented Jun 17, 2021

github-actions bot commented Jun 18, 2021

AWS Security Hub parser add unique_id_from_tool and deduplication #4376

AWS Security Hub parser add unique_id_from_tool and deduplication #4376

Conversation

nlandais commented Apr 27, 2021 • edited Loading

valentijnscholten left a comment

Choose a reason for hiding this comment

damiencarol left a comment

Choose a reason for hiding this comment

nlandais commented May 18, 2021

damiencarol left a comment

Choose a reason for hiding this comment

damiencarol commented May 22, 2021

damiencarol commented Jun 16, 2021

github-actions bot commented Jun 17, 2021

github-actions bot commented Jun 18, 2021

nlandais commented Apr 27, 2021 •

edited

Loading