rhel: add csaf/vex updater #1165

crozzy · 2023-11-29T22:08:46Z

TODO

codecov · 2023-11-29T22:17:37Z

Codecov Report

Attention: Patch coverage is 55.02959% with 228 lines in your changes are missing coverage. Please review.

Project coverage is 56.25%. Comparing base (4c5aa9e) to head (fbdb2b9).
Report is 1 commits behind head on main.

Files	Patch %	Lines
rhel/vex/parser.go	61.34%	70 Missing and 22 partials ⚠️
rhel/vex/fetcher.go	52.24%	51 Missing and 34 partials ⚠️
rhel/vex/updater.go	46.03%	28 Missing and 6 partials ⚠️
rhel/matcherfactory.go	0.00%	10 Missing ⚠️
rhel/matcher.go	46.15%	6 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #1165    +/-   ##
========================================
  Coverage   56.24%   56.25%            
========================================
  Files         266      270     +4     
  Lines       16838    17336   +498     
========================================
+ Hits         9471     9752   +281     
- Misses       6402     6558   +156     
- Partials      965     1026    +61

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

rhel/vex/fetcher.go

rhel/vex/parser.go

crozzy · 2024-04-29T18:20:35Z

I figure this is probably ready for a review even if it can't be fully merged yet

rhel/matcher.go

RTann · 2024-05-02T22:02:41Z

updater/defaults/defaults.go

@@ -63,6 +64,7 @@ func inner(ctx context.Context) error {
 	updater.Register("debian", df)

 	updater.Register("osv", new(osv.Factory))
+	updater.Register("rhel-vex", new(vex.Factory))


Is the idea to support both OVAL and VEX at the same time for a period, or will this PR remove "rhel", too, later?

Probably remove the OVAL in the same PR

@RTann check out the last commit

just curious about the ordering here. is it pretty arbitrary?

Replace the Red Hat OVALv2 update source with the Red Hat CSAF/VEX data. Signed-off-by: crozzy <joseph.crosland@gmail.com>

Start matching repository CPEs based on the CPE subset relation. This change interprets VEX CPEs identifying Red Hat repositories as CPE matching expressions and looks for a subset relation with the record's repositoty CPE. Signed-off-by: crozzy <joseph.crosland@gmail.com>

Previously the IgnoreUnpatched config key was a part of the RHEL updater and would dictate whether or not the updater would ingest unpatched vulnerabilities. This change moves that key to the RHEL matcher and dictates whether the matcher should check for a fixed_in_version when querying potential vulnerabilities. This makes the config option more usable at the expense of DB size. Signed-off-by: crozzy <joseph.crosland@gmail.com>

Given that the rhel-vex data will be responsible for Red Hat vulnerabilities we no longer want the existing OVAL updater to be a default (or even selectable). This patch also removes existing RHEL OVAL data from the matcher DB. Signed-off-by: crozzy <joseph.crosland@gmail.com>

RTann · 2024-05-16T21:39:26Z

datastore/postgres/migrations/matcher/13-delete-rhel-oval.sql

+-- The rhel-vex updater will now be responsible for RHEL advisories so we have
+-- to delete the existing rhel vulnerabilities.
+DELETE FROM update_operation WHERE updater ~ 'RHEL[5-9]-*';
+DELETE FROM vuln v2 where v2.updater ~ 'RHEL[5-9]-*';


just curious why call this v2?

RTann · 2024-05-16T21:58:16Z

datastore/postgres/querybuilder.go

@@ -70,6 +70,8 @@ func buildGetQuery(record *claircore.IndexRecord, opts *datastore.GetOpts) (stri
 			ex = goqu.Ex{"dist_arch": record.Distribution.Arch}
 		case driver.RepositoryName:
 			ex = goqu.Ex{"repo_name": record.Repository.Name}
+		case driver.HasFixedInVersion:
+			ex = goqu.Ex{"fixed_in_version": goqu.Op{"neq": ""}}


just curious if this is preferred over something like exp.NeqOp.String(). This is wordier, but protects against mistakes and future-proofs it (though I doubt it'll change, and it looks correct go me)

I guess the unit test also proves this works and can future-proof it

RTann · 2024-05-16T22:00:50Z

internal/matcher/controller.go

@@ -142,7 +142,7 @@ func (mc *Controller) filter(ctx context.Context, interested []*claircore.IndexR
 		if err != nil {
 			return nil, err
 		}
-		filtered[record.Package.ID] = match
+		filtered[record.Package.ID] = append(filtered[record.Package.ID], match...)


hard to tell without context: what was this overwriting before? How bad was this?

RTann · 2024-05-16T22:05:31Z

rhel/matcher.go

-		driver.PackageModule,
-		driver.RepositoryName,
+func (m *Matcher) Query() []driver.MatchConstraint {
+	mcs := []driver.MatchConstraint{driver.PackageModule}


curious why do we check the package module? does every package have a related module? why not check for distribution name?

RTann · 2024-05-16T22:09:41Z

rhel/matcher.go

+// Vulnerable() will interpret the claircore.Vulnerability.Repo.CPE
+// as a CPE match expression and in order to be considered vulnerable
+// the relationship between claircore.IndexRecord.Repository.CPE and
+// the claircore.Vulnerability.Repo.CPE needs to be a CPE Name Comparison
+// Relation of SUBSET(⊂)(source is a subset of, or equal to the target).
+// https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7696.pdf Section 6.2.


Suggested change

// Vulnerable() will interpret the claircore.Vulnerability.Repo.CPE

// as a CPE match expression and in order to be considered vulnerable

// the relationship between claircore.IndexRecord.Repository.CPE and

// the claircore.Vulnerability.Repo.CPE needs to be a CPE Name Comparison

// Relation of SUBSET(⊂)(source is a subset of, or equal to the target).

// https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7696.pdf Section 6.2.

// Vulnerable will interpret the claircore.Vulnerability.Repo.CPE

// as a CPE match expression, and to be considered vulnerable,

// the relationship between claircore.IndexRecord.Repository.CPE and

// the claircore.Vulnerability.Repo.CPE needs to be a CPE Name Comparison

// Relation of SUBSET(⊂)(source is a subset of, or equal to the target).

// https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7696.pdf Section 6.2.

RTann · 2024-05-16T23:32:02Z

rhel/vex/fetcher.go

+			if err != nil {
+				return nil, hint, fmt.Errorf("error making advisory request %w", err)
+			}
+			if res.StatusCode != http.StatusOK {


missing a Body.Close() when we reach this

RTann · 2024-05-16T23:33:03Z

rhel/vex/fetcher.go

+			buf.Reset()
+			bc.Reset()
+			l++
+			res.Body.Close()


another option is to put this into a separate func so we can defer without worrying about the loop

RTann · 2024-05-16T23:34:47Z

rhel/vex/fetcher.go

+				return nil, hint, fmt.Errorf("unexpected response from advisary URL: %s %s", res.Status, req.URL)
+			}
+
+			_, err = buf.ReadFrom(res.Body)


so does this cover the case where this is the first time we read the VEX data and there is a newer version of a few files? This will overwrite data for those files?

RTann · 2024-05-16T23:37:42Z

rhel/vex/parser.go

+)
+
+// Parse implements [driver.Updater].
+func (u *VEXUpdater) Parse(ctx context.Context, contents io.ReadCloser) ([]*claircore.Vulnerability, error) {


For StackRox's usage of Claircore at the moment, we pretty much start fresh every time. Does not implementing Parse mean our use case may not work? Related: will the jsonblob stuff work OOTB?

RTann · 2024-05-16T23:37:52Z

rhel/vex/parser.go

Will need to look at this another time

crozzy force-pushed the csaf-vex branch from 6c09465 to a3be9dd Compare November 29, 2023 22:12

crozzy force-pushed the csaf-vex branch from a3be9dd to a345799 Compare November 30, 2023 21:59

crozzy force-pushed the csaf-vex branch 7 times, most recently from a834c67 to 0c787f2 Compare December 21, 2023 22:36

crozzy force-pushed the csaf-vex branch 7 times, most recently from 758d6a4 to fbfbf82 Compare January 4, 2024 20:04

crozzy force-pushed the csaf-vex branch 4 times, most recently from 89006e1 to 4deb036 Compare January 25, 2024 17:59

crozzy force-pushed the csaf-vex branch 5 times, most recently from 12e41cb to 052599d Compare February 2, 2024 22:42

hdonnay reviewed Feb 5, 2024

View reviewed changes

rhel/vex/fetcher.go Outdated Show resolved Hide resolved

hdonnay reviewed Feb 5, 2024

View reviewed changes

rhel/vex/fetcher.go Outdated Show resolved Hide resolved

hdonnay reviewed Feb 5, 2024

View reviewed changes

rhel/vex/parser.go Outdated Show resolved Hide resolved

crozzy force-pushed the csaf-vex branch from 9d9e53a to cb43f01 Compare February 5, 2024 21:47

crozzy force-pushed the csaf-vex branch 6 times, most recently from e30a749 to abfa0e0 Compare April 29, 2024 18:19

crozzy requested a review from hdonnay April 29, 2024 18:20

crozzy force-pushed the csaf-vex branch from abfa0e0 to 49217ca Compare May 1, 2024 21:59

RTann reviewed May 2, 2024

View reviewed changes

rhel/matcher.go Show resolved Hide resolved

RTann reviewed May 2, 2024

View reviewed changes

crozzy force-pushed the csaf-vex branch 8 times, most recently from bf52b0b to ab36dfd Compare May 10, 2024 16:17

crozzy force-pushed the csaf-vex branch 5 times, most recently from 0896bbd to 818d85d Compare May 14, 2024 22:13

crozzy added 4 commits May 14, 2024 15:22

rhel: add csaf/vex updater

162d977

Replace the Red Hat OVALv2 update source with the Red Hat CSAF/VEX data. Signed-off-by: crozzy <joseph.crosland@gmail.com>

crozzy force-pushed the csaf-vex branch from 818d85d to fbdb2b9 Compare May 14, 2024 22:42

RTann reviewed May 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rhel: add csaf/vex updater #1165

rhel: add csaf/vex updater #1165

crozzy commented Nov 29, 2023 •

edited

codecov bot commented Nov 29, 2023 •

edited

crozzy commented Apr 29, 2024

RTann May 2, 2024

crozzy May 2, 2024

crozzy May 14, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

RTann May 16, 2024

rhel: add csaf/vex updater #1165

Are you sure you want to change the base?

rhel: add csaf/vex updater #1165

Conversation

crozzy commented Nov 29, 2023 • edited

codecov bot commented Nov 29, 2023 • edited

Codecov Report

crozzy commented Apr 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crozzy commented Nov 29, 2023 •

edited

codecov bot commented Nov 29, 2023 •

edited