Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhanced data schema for Cloud-native Workload Vulnerability Data Storage for Analytics, Reporting and Policy Evaluation. #145

Merged
merged 4 commits into from May 25, 2021
Merged

Enhanced data schema for Cloud-native Workload Vulnerability Data Storage for Analytics, Reporting and Policy Evaluation. #145

merged 4 commits into from May 25, 2021

Conversation

prahaladdarkin
Copy link
Contributor

This proposal will lay the foundational work that allow advanced reporting, analytics, policy evaluation and integration with third-party systems providing these capabilities such as Open Policy Agent.
The contents of this proposal are based on the document describing the Cloud Native Workload - Common Vulnerability Evaluation and Reporting Framework.

@danielpacak danielpacak self-requested a review June 26, 2020 08:18
@steven-zou steven-zou added area/interrogation-service Services like vulnerability scanning and compliance checking etc. kind/proposal labels Jul 1, 2020
prahaladdarkin
prahaladdarkin commented Jul 30, 2020
View reviewed changes
Copy link
Contributor Author

@prahaladdarkin prahaladdarkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed addressing of all review comments. @danielpacak please take a look and let us know if all looks good

proposals/new/common_vulnerability_schema_proposal.md Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
danielpacak
danielpacak previously approved these changes Aug 17, 2020
View reviewed changes
Copy link
Contributor

@danielpacak danielpacak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

steven-zou
steven-zou previously approved these changes Aug 27, 2020
View reviewed changes
Copy link
Contributor

@steven-zou steven-zou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@prahaladdarkin

I think this is the right way to handle the vuln data. With this schema refactor, it's very friendly to do ad-hoc query that is a feature we're planning to support later.

@prahaladdarkin
Copy link
Contributor Author

@danielpacak and @steven-zou - one query.. as i was working on a PoC to convert the raw report to the normalized schema, I see that the current table scan_report and the new schema (scan_report_v2) in this proposal are almost identical except that the new proposed table does not have the Report column.
So wanted to understand the common practice followed - would it be OK to re-use the original scan_report table itself instead of creating a new table again. IMO, we can re-use the existing table. This way we can allow the following:

  • Allow the existing functionality and schema to co-exist with the new ad-hoc query functionality that will be introduced until we switch over completely to the new functionality.
  • Reduce the amount of code change - all through the various layers right up to the REST APIs, we would not need to change the name of the table
  • Avoid complex queries and data management - copying the content from existing scan_report to the new scan_report_v2, deleting and cleanup.

Let me know your views

@prahaladdarkin prahaladdarkin mentioned this pull request Sep 30, 2020
Merged
@prahaladdarkin prahaladdarkin mentioned this pull request Nov 26, 2020
Merged
prahaladdarkin added a commit to prahaladdarkin/harbor that referenced this pull request Dec 2, 2020
@prahaladdarkin
Store vulnerability report from scanner into a relational format
145cb5a 
Convert vulnerability report JSON obtained  from scanner into a relational format describe in:goharbor/community#145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
prahaladdarkin added a commit to prahaladdarkin/harbor that referenced this pull request Dec 2, 2020
@prahaladdarkin
Store vulnerability report from scanner into a relational format
095af42 
Convert vulnerability report JSON obtained  from scanner into a relational format describe in:goharbor/community#145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
prahaladdarkin added a commit to prahaladdarkin/harbor that referenced this pull request Dec 6, 2020
@prahaladdarkin
Store vulnerability report from scanner into a relational format
e5bafb0 
Convert vulnerability report JSON obtained  from scanner into a relational format describe in:goharbor/community#145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
prahaladdarkin added a commit to prahaladdarkin/harbor that referenced this pull request Dec 18, 2020
@prahaladdarkin
Store vulnerability report from scanner into a relational format
6030179 
Convert vulnerability report JSON obtained  from scanner into a relational format describe in:goharbor/community#145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
prahaladdarkin added a commit to prahaladdarkin/harbor that referenced this pull request Dec 18, 2020
@prahaladdarkin
Store vulnerability report from scanner into a relational format
143ff9e 
Convert vulnerability report JSON obtained  from scanner into a relational format describe in:goharbor/community#145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
prahaladdarkin added a commit to prahaladdarkin/harbor that referenced this pull request Dec 23, 2020
@prahaladdarkin
Store vulnerability report from scanner into a relational format
8ed0d9b 
Convert vulnerability report JSON obtained  from scanner into a relational format describe in:goharbor/community#145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
prahaladdarkin added a commit to prahaladdarkin/harbor that referenced this pull request Dec 23, 2020
@prahaladdarkin
Store vulnerability report from scanner into a relational format
2cfec73 
Convert vulnerability report JSON obtained  from scanner into a relational format describe in:goharbor/community#145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
heww pushed a commit to goharbor/harbor that referenced this pull request Dec 25, 2020
@prahaladdarkin
Store vulnerability data from scanner into a relational format (#13616)
a890b28 
feat: Store vulnerability report from scanner into a relational format

Convert vulnerability report JSON obtained  from scanner into a relational format describe in:goharbor/community#145

Signed-off-by: prahaladdarkin <prahaladd@vmware.com>
@prahaladdarkin
Proposal for migration of software vulnerability data to a normalized…
b6ae36b 
… schema

Signed-off-by: prahaladd <prahaladd@vmware.com>
danielpacak
danielpacak suggested changes Jan 4, 2021
View reviewed changes
Copy link
Contributor

@danielpacak danielpacak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👋 @prahaladdarkin I left a few comments to tidy up this proposal and align it with the actual Go code.

proposals/new/common_vulnerability_schema_proposal.md Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
@prahaladdarkin
Incorporating review comments and adding database diagram
997cdc3 
Signed-off-by: prahaladd <prahaladd@vmware.com>
prahaladdarkin
prahaladdarkin commented Jan 6, 2021
View reviewed changes
Copy link
Contributor Author

@prahaladdarkin prahaladdarkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Incorporated all review comments and resolved outstanding conversations.

proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Outdated Show resolved Hide resolved
proposals/new/common_vulnerability_schema_proposal.md Show resolved Hide resolved
steven-zou
steven-zou approved these changes Jan 8, 2021
View reviewed changes
Copy link
Contributor

@steven-zou steven-zou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The feature will be delivered in V2.2.

@zhill zhill mentioned this pull request Jan 24, 2021
Open
@steven-zou steven-zou merged commit ebf3241 into goharbor:master May 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reasonerjt reasonerjt reasonerjt left review comments

@danielpacak danielpacak danielpacak approved these changes

@steven-zou steven-zou steven-zou approved these changes

Assignees
No one assigned
Labels
area/interrogation-service Services like vulnerability scanning and compliance checking etc. kind/proposal
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@prahaladdarkin @danielpacak @reasonerjt @steven-zou