Configuration groundwork #182

hdonnay · 2020-05-26T17:20:48Z

These configuration knobs lay the groundwork for having scanners that are configurable and make RPCs, and for making updaters that are configurable.

I'd be fine with dropping the alpine commit out of this stack, it's there as a PoC.

ldelossa

I'd like to see the alpine and aws config ripped out, as changing the config values will have no effect after a layer was scanned.

I think Updater configs need some more thought, do we want a config that looks like?

UpdaterConfigs:
  rhel4:
    ...
  rhel5:
    ...
  rhel6:
   ...
  rhel7:
   ...
  ubuntu-disco:
   ...
 etc...

I'd lean toward using the UpdaterSet names that we began using to configure which Updaters runs

UpdaterConfigs:
  ubuntu:
    ...
  rhel:
    ...
  oracle
    ...

ldelossa · 2020-05-27T20:29:54Z

internal/indexer/ecosystem.go

 			}
+			seen[n] = struct{}{}
+			if _, ok := s.(RPCScanner); airgap && ok {


can we add an INFO() log to each scanner name which is excluded due to airgap?

ldelossa · 2020-05-27T20:30:54Z

internal/indexer/layerscanner/layerscanner.go

@@ -69,7 +70,7 @@ func (ls *layerScanner) Scan(ctx context.Context, manifest claircore.Digest, lay

 	ls.cc = make(chan struct{}, ccMin)

-	ps, ds, rs, err := indexer.EcosystemsToScanners(ctx, ls.Opts.Ecosystems)
+	ps, ds, rs, err := indexer.EcosystemsToScanners(ctx, ls.Opts.Ecosystems, ls.Opts.Airgap)


part of me wants the airgap filtering to be it's own method and not be done in EcosystemsToScanner as it'll read a little more declaratively in the construction of the Indexer. what do you think?

ldelossa · 2020-05-27T20:48:59Z

libvuln/init.go

@@ -28,6 +29,12 @@ func initUpdaters(ctx context.Context, opts *Opts, db *sqlx.DB, store vulnstore.
 			eC <- fmt.Errorf("duplicate updater found in UpdaterFactory. all names must be unique: %s", u.Name())
 			return
 		}
+		if c, ok := u.(driver.Configurable); ok {


I think this is going to take a little more thinking about. This is going to be an issue where we generate the names dynamically like Oracle (generating the date).

ldelossa · 2020-05-27T20:50:47Z

internal/indexer/versionedscanner.go

@@ -15,6 +21,23 @@ type VersionedScanner interface {
 	Kind() string
 }

+// ConfigDeserializer can be thought of as a curried Unmarshal function.


Suggested change

// ConfigDeserializer can be thought of as a curried Unmarshal function.

// ConfigDeserializer can be thought of as an Unmarshal function.

don't believe curried is the right word here as it's just a closure over another method.

Unmarshal usually has the signature func([]byte, interface{}) error, so I was trying to point out it's that same concept with the []byte closed over from a different scope.

ldelossa · 2020-05-27T20:51:17Z

libvuln/driver/updater.go

 type Fingerprint string
+
+// ConfigUnmarshaler can be thought of as a curried Unmarshal function, or a


Suggested change

// ConfigUnmarshaler can be thought of as a curried Unmarshal function, or a

// ConfigUnmarshaler can be thought of as an Unmarshal function, or a

same

ldelossa · 2020-05-27T20:53:16Z

libvuln/opts.go

@@ -59,6 +60,11 @@ type Opts struct {
 	//
 	// This list will me merged with the default matchers.
 	Matchers []driver.Matcher
+	// Configs is a map of functions for configuration of Updaters.
+	Configs map[string]driver.ConfigUnmarshaler


May as well call this UpdaterConfigs if we ever need to pass Matcher configs down the line.

ldelossa · 2020-05-29T17:55:25Z

internal/indexer/layerscanner/layerscanner.go

+	//
+	// This is structured into a closure so that we can type-erase to the
+	// common VersionedScanner interface and only write all this logic once.
+	filter := func(s indexer.VersionedScanner) bool {


Is there any reason this needs to be done at Scan time and not on construction of the layer scanner?

I don't think so.

ldelossa · 2020-05-29T17:56:37Z

internal/indexer/layerscanner/layerscanner.go

+	//
+	// This is structured into a closure so that we can type-erase to the
+	// common VersionedScanner interface and only write all this logic once.
+	filter := func(s indexer.VersionedScanner) bool {


large inline function definitions make it pretty tough to understand the gist of the Scan mechanics.
Can this function and the loops below be created as methods with a declarative name so the mechanics of the semaphore usage and concurrent scanning be the focus?

internal/indexer/layerscanner/layerscanner.go

ldelossa · 2020-05-29T18:07:15Z

internal/indexer/layerscanner/layerscanner.go

+	g, ctx := errgroup.WithContext(ctx)
+	// Scan is a closure to capture the values in the loops and then call the
+	// scan method.
+	scan := func(l *claircore.Layer, s indexer.VersionedScanner) func() error {


can you avoid this scan variable shadow with ls.scan?
put the sem on the layerScanner struct and make it sharable and then use it in ls.scan
seems a bit silly to have ls.scan and then a scan closure just to capture a sem that can be shared in the ls.scan method anyway.

It's not to capture the semaphore, it's to capture the loop variables.

can you make it so the process of performing a scan is self contained in layerScanner.scan?
its simpler then trying to hop around between the Scan function, the scan variable, the function that is returned by the scan variable, the call to the function that is returned by the scan variable, and the ls.scan function.

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

This exploded a bit into also refactoring the layerscanner, which is the primary place where scanners are actually invoked. Signed-off-by: Hank Donnay <hdonnay@redhat.com>

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

hdonnay requested a review from ldelossa May 26, 2020 17:20

hdonnay requested review from alecmerdler and jzelinskie as code owners May 26, 2020 17:20

hdonnay force-pushed the hank/remote-scanner branch from c25e066 to 9f85618 Compare May 26, 2020 17:27

ldelossa suggested changes May 27, 2020

View reviewed changes

hdonnay force-pushed the hank/remote-scanner branch 5 times, most recently from de2f48a to c6c1ba0 Compare May 29, 2020 18:00

ldelossa suggested changes May 29, 2020

View reviewed changes

hdonnay force-pushed the hank/remote-scanner branch from c6c1ba0 to 263f8cc Compare May 29, 2020 18:53

ldelossa previously approved these changes May 29, 2020

View reviewed changes

hdonnay dismissed ldelossa’s stale review via 86978e1 May 29, 2020 20:37

hdonnay force-pushed the hank/remote-scanner branch from 263f8cc to 86978e1 Compare May 29, 2020 20:37

ldelossa previously approved these changes May 29, 2020

View reviewed changes

indexer: add Configurable interface

11b4676

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

hdonnay dismissed ldelossa’s stale review via 7518c5f May 29, 2020 21:04

hdonnay force-pushed the hank/remote-scanner branch from 86978e1 to 7518c5f Compare May 29, 2020 21:04

hdonnay added 2 commits May 29, 2020 19:10

libindex: use new Configurable interfaces

ac10351

This exploded a bit into also refactoring the layerscanner, which is the primary place where scanners are actually invoked. Signed-off-by: Hank Donnay <hdonnay@redhat.com>

aws: ensure Close call gets to underlying File

79bad1e

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

hdonnay force-pushed the hank/remote-scanner branch from 7518c5f to 79bad1e Compare May 29, 2020 23:12

ldelossa approved these changes Jun 1, 2020

View reviewed changes

hdonnay merged commit 79bad1e into master Jun 1, 2020

hdonnay deleted the hank/remote-scanner branch June 1, 2020 13:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuration groundwork #182

Configuration groundwork #182

hdonnay commented May 26, 2020

ldelossa left a comment

ldelossa May 27, 2020

ldelossa May 27, 2020

ldelossa May 27, 2020

ldelossa May 27, 2020

hdonnay May 28, 2020

ldelossa May 27, 2020

ldelossa May 27, 2020

ldelossa May 29, 2020

hdonnay May 29, 2020

ldelossa May 29, 2020

ldelossa May 29, 2020 •

edited

hdonnay May 29, 2020

ldelossa May 29, 2020 •

edited

	// ConfigDeserializer can be thought of as a curried Unmarshal function.
	// ConfigDeserializer can be thought of as an Unmarshal function.

		type Fingerprint string

		// ConfigUnmarshaler can be thought of as a curried Unmarshal function, or a

Configuration groundwork #182

Configuration groundwork #182

Conversation

hdonnay commented May 26, 2020

ldelossa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ldelossa May 29, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ldelossa May 29, 2020 • edited

Choose a reason for hiding this comment

ldelossa May 29, 2020 •

edited

ldelossa May 29, 2020 •

edited