Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrating new scanner Whatweb to the secureCodeBox (closes #567) #592

Merged
merged 56 commits into from
Sep 1, 2021
Merged

Integrating new scanner Whatweb to the secureCodeBox (closes #567) #592

merged 56 commits into from
Sep 1, 2021

Conversation

SebieF
Copy link
Contributor

@SebieF SebieF commented Aug 11, 2021 •
edited

Description

This PR adds the new fingerprinting scanner Whatweb to the scb. It identifies websites and recognizes used technology. This can be especially useful for cascading scans.

Checklist

  • Test your changes as thoroughly as possible before you commit them. Preferably, automate your test by unit/integration tests.
  • Make sure npm test runs for the whole project.
  • Make codeclimate checks happy

(closes #567)

@SebieF SebieF added scanner Implement or update a security scanner testing Improvements or additions regarding the test setup labels Aug 11, 2021
@SebieF SebieF self-assigned this Aug 11, 2021
@SebieF SebieF added this to In progress in secureCodeBox v3 via automation Aug 11, 2021
J12934 and others added 21 commits August 11, 2021 20:18
@J12934 @SebieF
Update Client to include fix
c2890ef 
Signed-off-by: Jannik Hollenbach <jannik.hollenbach@iteratec.com>
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@J12934 @SebieF
Update ObjectMappers in code base to also register modules
66e600b 
Signed-off-by: Jannik Hollenbach <jannik.hollenbach@iteratec.com>
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@J12934 @SebieF
Fix additional crash caused by new validation of severities in Defect…
7d30ba2 
…Dojo

DefectDojo apparently now validates that these are in the right format, nice 👍

Signed-off-by: Jannik Hollenbach <jannik.hollenbach@iteratec.com>
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding scanner Dockerfile
d95432d 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding whatweb parser
8f2994c 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding docs folder
ed0d8a3 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding template files (wip)
46d672b 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating helmignore
be56dc9 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Removing unnecessary template files
f465916 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating helm files
ececf46 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Changing parser test description
fc6cc1b 
Adding securecodebox.io

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding helm template files
5b1baeb 
Scan type, Parse definition, Cascading rules

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding example cascading rule
309e30b 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding helmdocs (wip)
88932f0 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding integration test for whatweb
38f9a17 
Basic scanning of example.com and check for invalid argument

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating ci
4199505 
Adding parser, scanner and integration test

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating Helm Docs
16af102 
Signed-off-by: GitHub Actions <securecodebox@iteratec.com>
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating helm docs readme
7d09bb0 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating Helm Docs
951dbc8 
Signed-off-by: GitHub Actions <securecodebox@iteratec.com>
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating Helm Docs
b004f92 
Signed-off-by: GitHub Actions <securecodebox@iteratec.com>
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding cascading-rules hint to helm docs
e8940b7 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
SebieF and others added 4 commits August 11, 2021 18:21
@SebieF @secureCodeBoxBot
Updating Helm Docs
ca684a6 
Signed-off-by: GitHub Actions <securecodebox@iteratec.com>
@SebieF
Fixing typo in Chart.yaml
d465fe0 
Writing Whatweb with Upper case

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Merge remote-tracking branch 'origin/scanner/whatweb' into scanner/wh…
b94940b 
…atweb
@SebieF @secureCodeBoxBot
Updating Helm Docs
b662bea 
Signed-off-by: GitHub Actions <securecodebox@iteratec.com>
@SebieF
Replacing ifs with switch
0260cab 
Making codeclimate happy

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Making codeclimate happy
3547972 
At least I tried

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF SebieF marked this pull request as ready for review August 25, 2021 14:34
J12934
J12934 requested changes Aug 27, 2021
View reviewed changes
Copy link
Member

@J12934 J12934 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work 👍
Only some small things, other than that, the PR should be ready to be merged

scanners/whatweb/scanner/Dockerfile Outdated Show resolved Hide resolved
scanners/whatweb/parser/package.json Outdated Show resolved Hide resolved
scanners/whatweb/parser/Dockerfile Outdated Show resolved Hide resolved
SebieF and others added 10 commits August 28, 2021 16:43
@SebieF
Removing unnecessary package files
959d29e 
Parser has no dependencies

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Simplifying Dockerfile for Parser
58a7a8c 
Removed package.json dependencies

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating scanner Dockerfile to use latest version from git
ffcd8fb 
Now using ruby image, installing whatweb manually via make. Upgrades from version 0.4.9 on Debian to 0.5.5

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adapting parser tests to latest scanner version
2dd70a1 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Removing pull policy
8d4903b 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adding usergroup to scanner Dockerfile
3ec8dc9 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Updating scanner version
05f702a 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF @secureCodeBoxBot
Updating Helm Docs
3f8cbed 
Signed-off-by: GitHub Actions <securecodebox@iteratec.com>
@SebieF
Restoring pull policy
d30c0b6 
Failing integration tests otherwise

Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Merge remote-tracking branch 'origin/scanner/whatweb' into scanner/wh…
aae6214 
…atweb
@SebieF SebieF requested a review from J12934 August 28, 2021 16:12
@SebieF
Adding parse-definition consistency
3fe0348 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
@SebieF
Adjusting consistency of parse-definition
60f5325 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>
rseedorff
rseedorff previously approved these changes Aug 31, 2021
View reviewed changes
@rseedorff
Copy link
Member

Hi @SebieF,
Great work 👍
One thing last: your PR has a conflict that needs to be solved... than the PR should be ready to be merged

@SebieF SebieF requested a review from rseedorff August 31, 2021 14:21
@SebieF @rseedorff
Adding parser validation in ci
a990aa9 
Signed-off-by: Sebastian <sebastian.franz@iteratec.com>

Co-authored-by: Robert Seedorff <Robert.Seedorff@iteratec.com>
@Weltraumschaf Weltraumschaf merged commit eddd469 into main Sep 1, 2021
secureCodeBox v3 automation moved this from To Review to Done Sep 1, 2021
@Weltraumschaf Weltraumschaf deleted the scanner/whatweb branch September 1, 2021 10:42
@SebieF SebieF moved this from Done to counter in secureCodeBox v3 Sep 3, 2021
@snyk-bot snyk-bot mentioned this pull request Sep 27, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rseedorff rseedorff rseedorff approved these changes

@J12934 J12934 Awaiting requested review from J12934

Assignees

@SebieF SebieF

Labels
scanner Implement or update a security scanner testing Improvements or additions regarding the test setup
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add WhatWeb as new scanner for fingerprinting usecases
4 participants
@SebieF @J12934 @rseedorff @Weltraumschaf