config: reorganize updater configuration

In forthcoming commits, additional processes will need to read updater
configurations for running updaters. This moves updaters' configuration
to a top-level key and documents them.

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

Documentation/operation.md

@@ -104,3 +104,104 @@ auth:
       MDQ4ODBlNDAtNDc0ZC00MWUxLThhMzAtOTk0MzEwMGQwYTMxCg==
     iss: 'issuer'
 ```
+
+## Updaters
+
+Clair utilizes go packages we call "updaters" that encapsulate the logic of
+fetching and parsing different vulnerability databases. Updaters are usually
+pared with a matcher to interpret if and how any vulnerability is related to a
+package.
+
+Operators may wish to update the vulnerability database less frequently or not
+import vulnerabilities from databases that they know will not be used.
+
+### Configuration
+
+Updaters can be configured by `updaters` key at the top of the configuration. If
+updaters are being run automatically within the matcher processes, as is the
+default, the period for running updaters is configured under the matcher's
+configuration stanza.
+
+#### Choosing Sets
+
+Specific sets of updaters can be selected by the `sets` list. If not present,
+the defaults of all upstream updaters will be used.
+
+```yaml
+updaters:
+  sets:
+    - rhel
+```
+
+#### Filtering Updaters
+
+To disallow an updater from running without disabling an entire set, the filter
+option can be used. The provided string will be interpreted as a go [regexp]
+used to disallow any updater with a name that does not match. **Note:** This
+means that an empty string matches *any* string, not no strings.
+
+```yaml
+updaters:
+  filter: '^$'
+```
+
+#### Specific Updaters
+
+Configuration for specific updaters can be passed by putting a key underneath
+the `config` member of the `updaters` object. The name of an updater may be
+constructed dynamically; users should examine logs to double-check names.
+The specific object that an updater expects should be covered in the updater's
+documentation.
+
+For example, to have the "rhel" updater fetch a manifest from a different
+location:
+
+```yaml
+updaters:
+  config:
+    rhel:
+      url: https://example.com/mirror/oval/PULP_MANIFEST
+```
+
+### Airgap
+
+For additional flexibility, Clair supports running updaters in a different
+environment and importing the results. This is aimed at supporting installations
+that disallow the Clair cluster from talking to the Internet directly. An update
+procedure needs to arrange to call the relevant `clairctl` command in an
+environment with access to the Internet, move the resulting artifact across the
+airgap according to site policy, and then call the relevant `clairctl` command
+to import the updates.
+
+For example:
+
+```sh
+# On a workstation, run:
+clairctl updater-export updates.gz
+```
+
+```sh
+# Move the resulting file to a place reachable by the cluster:
+scp updates.gz internal-webserver:/var/www/
+```
+
+```sh
+# On a pod inside the cluster, import the file:
+clairctl updater-import http://web.svc/updates.gz
+```
+
+#### Configuration
+
+Matcher processes should have the `disable_updaters` key set to disable
+automatic updaters running.
+
+```yaml
+matcher:
+  disable_updaters: true
+```
+
+Desired updaters should be selected by the normal configuration mechanism.
+
+## Indexers
+
+#### Configuration

config/config.go

@@ -4,6 +4,9 @@ import (
 	"fmt"
 	"net/url"
 	"strings"
+
+	"github.com/quay/claircore/libvuln/driver"
+	"gopkg.in/yaml.v3"
 )
 
 // Clair Modes
@@ -55,6 +58,51 @@ type Config struct {
 	Auth     Auth     `yaml:"auth" json:"auth"`
 	Trace    Trace    `yaml:"trace" json:"trace"`
 	Metrics  Metrics  `yaml:"metrics" json:"metrics"`
+	Updaters Updaters `yaml:"updaters" json:"updaters"`
+}
+
+// Updaters configures updater behavior.
+type Updaters struct {
+	// A slice of strings representing which
+	// updaters will be used.
+	//
+	// If nil all default UpdaterSets will be used
+	//
+	// The following sets are supported by default:
+	// "alpine"
+	// "aws"
+	// "debian"
+	// "oracle"
+	// "photon"
+	// "pyupio"
+	// "rhel"
+	// "suse"
+	// "ubuntu"
+	Sets []string `yaml:"sets" json:"sets"`
+	// Config holds configuration blocks for UpdaterFactories and Updaters,
+	// keyed by name.
+	//
+	// These are defined by the updater implementation and can't be documented
+	// here. Improving the documentation for these is an open issue.
+	Config map[string]yaml.Node `yaml:"config" json:"config"`
+	// Filter is a regexp that disallows updaters that do not match from
+	// running.
+	Filter string `yaml:"filter" json:"filter"`
+}
+
+func (u *Updaters) FilterSets(m map[string]driver.UpdaterSetFactory) {
+	if u.Sets != nil {
+	Outer:
+		for k := range m {
+			for _, n := range u.Sets {
+				if k == n {
+					continue Outer
+				}
+			}
+			delete(m, k)
+		}
+	}
+	return
 }
 
 // Validate confirms the necessary values to support

config/matcher.go

@@ -1,5 +1,7 @@
 package config
 
+import "time"
+
 type Matcher struct {
 	// A Postgres connection string.
 	//
@@ -23,20 +25,13 @@ type Matcher struct {
 	//
 	// Whether Matcher nodes handle migrations to their databases.
 	Migrations bool `yaml:"migrations" json:"migrations"`
-	// A slice of strings representing which
-	// updaters matcher will create.
+	// Period controls how often updaters are run.
 	//
-	// If nil all default UpdaterSets will be used
+	// The default is 30 minutes.
+	Period *time.Duration `yaml:"period" json:"period"`
+	// DisableUpdaters disables the updater's running of matchers.
 	//
-	// The following sets are supported:
-	// "alpine"
-	// "aws"
-	// "debian"
-	// "oracle"
-	// "photon"
-	// "pyupio"
-	// "rhel"
-	// "suse"
-	// "ubuntu"
-	UpdaterSets []string `yaml:"updater_sets" json:"updater_sets"`
+	// This should be toggled on if vulnerabilities are being provided by
+	// another mechanism.
+	DisableUpdaters bool `yaml:"disable_updaters" json:"disable_updaters"`
 }

local-dev/clair/config.yaml

@@ -2,6 +2,7 @@
 log_level: debug-color
 introspection_addr: ""
 http_listen_addr: ":6000"
+updaters: {}
 indexer:
   connstring: host=clair-db port=5432 user=clair dbname=clair sslmode=disable
   scanlock_retry: 10
@@ -25,12 +26,12 @@ notifier:
   amqp:
     direct: true
     exchange:
-        name: "" 
-        type: "direct"
-        durable: true
-        auto_delete: false
+      name: ""
+      type: "direct"
+      durable: true
+      auto_delete: false
     uris: ["amqp://guest:guest@clair-rabbitmq:5672/"]
-    routing_key: "notifications" 
+    routing_key: "notifications"
     callback: "http://clair-notifier/api/v1/notifications"
 # tracing and metrics config
 trace:
@@ -42,4 +43,3 @@ trace:
     service_name: "clair"
 metrics:
   name: "prometheus"
-