Operator

[hdonnay]

Problem

Clair's deployment is flexible and complex enough that we should provide help beyond documentation for operating Clair.

Solution

We should provide an Operator for managing Clair deployments.

This fits in well with the strategy for Quay (our biggest known integrator) and should generalize to any k8s environment.

Why not...

Helm

I do not think Helm takes a very sound approach, with the stringly typed yaml templating.

Questions

What should be supported?

I think we should only support a heterogeneous deployment, i.e. more than just Combo mode, so at least two deployments, one each for the matcher and indexer, and an optional other for the notifier.

• We need to develop metrics around how these are to be scaled.
• We need to figure out what configurables will be provided on the CRD.

Should we split out updater jobs into a cron job?

The updaters would then have their own cpu and memory accounting and have dashboard-level visibility.

What should be provided?

A database should probably be pulled in automatically.

[ldelossa]

Problem

Clair's deployment is flexible and complex enough that we should provide help beyond documentation for operating Clair.

Solution

We should provide an Operator for managing Clair deployments.

This fits in well with the strategy for Quay (our biggest known integrator) and should generalize to any k8s environment.

Why not...

Helm

I do not think Helm takes a very sound approach, with the stringly typed yaml templating.

Questions

What should be supported?

I think we should only support a heterogeneous deployment, i.e. more than just Combo mode, so at least two deployments, one each for the matcher and indexer, and an optional other for the notifier.

Since k8s pods are cheap and can be specified with custom resources, maybe its easiest to say "we deploy each service as a pod" - making the notifier pod not optional.

Each deployment should scale to some pod count independently.

Should we split out updater jobs into a cron job?

The updaters would then have their own cpu and memory accounting and have dashboard-level visibility.

Can this be accomplished without a design change ? I'd hesitate developing an operator while changing design and also during our app-sre integration.

What should be provided?

A database should probably be pulled in automatically.

Database should be provided. Do we want to support running a database per pod ?
We MUST support running against cloud databases as well, but that should fall mostly to our config.

[hdonnay]

I am not talking about multimaster or replication here. I'm only talking about the "microservice" proverb of "database per service". Running a postgres instance per "deployment" (I incorrectly said per-pod above).

I think we should provide one database engine, configured for 3 separate users and databases, possibly with each database on its own volume?
That seems like a middle ground between minimal moving parts and the ability for an operator to scale forward.

[ldelossa]

I am not talking about multimaster or replication here. I'm only talking about the "microservice" proverb of "database per service". Running a postgres instance per "deployment" (I incorrectly said per-pod above).

I think we should provide one database engine, configured for 3 separate users and databases, possibly with each database on its own volume?
That seems like a middle ground between minimal moving parts and the ability for an operator to scale forward.

I'm not exactly against this, but I do fear the operator's use in Quay.io where we expect high volume. If we wind up needing to split load on the databases - it would be nice if the operator allowed us a way to do this.

[hdonnay]

I was expecting that database to be unmanaged from the operator's perspective, and have it's own operator, human or otherwise.

[ldelossa]

Ohhhh I thought you were thinking about having the operator pull in the postgres database itself, this would technically allow a completely seamless deployment.

However, if we expect databases to be already setup, then the CRD would just take URI's to already managed and configured databases?

[hdonnay]

I think we should follow the quay operator's lead here and provide something if the user desires, but also be able to use existing/unmanaged resources.

[ldelossa]

We need to develop metrics around how these are to be scaled.

Will these metrics piggy back of our metrics work over the next month?

We need to figure out what configurables will be provided on the CRD.

• Pod counts per deployment (indexer, matcher, notifier)
• Field indicating whether to pull in a in-cluster DB or whether a Cloud Database URL is provided (if cloud DB URL is provided just forward this into generated config)
• Path routing mechanism to use (OpenShift Routes, Nginx, Traefik, etc...)
• Monitoring back end to utilize (Prometheus, Jaeger, what ever else we support,,,) - Operator will set-up the specified monitoring infrastructure and generate the necessary configuration values for Clair to utilize it.
• TLS Termination (we punt this to whoever is deploying Clair) - this maybe combined with Path routing solution but also it may not. For example OpenShift Ingresses do not lb internal traffic.
• The rest of our config options need to be exported in the CRD as well, such as UpdateInterval, UpdateRetention, etc...

[hdonnay]

Will these metrics piggy back of our metrics work over the next month?

I hope so. I think we need to look at the metrics of the various services under load and determine what are good values to use in an HPA. My gut instinct is that load and memory pressure are going to result in overprovisioning for the indexer, and metrics like request latency won't look "normal".

That reminds me, the metrics pass needs a way to denote API responses that did work vs served from the memoized result.

[hdonnay]

After some hacking and sketching, this is the plan:

ConfigMap/Secret driven:

Projecting our config into a CR means the operator would need the logic to create a config and duplicate any logic for version translations in the different versions of CRs and configs. It seems to me (and this seems to be the current preferred pattern) that referencing a ConfigMap or Secret containing a config keeps the operator's types much more tidy. For example, given the following object, the webhook could examine the indicated value, see it is invalid yaml, and reject the update to the ConfigMap.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: clair-config
  labels:
    projectquay.io/clair/validate-config: 'true'
  annotations:
    projectquay.io/clair/config: config.yaml
data:
  config.yaml: |
    ---
    matcher:
       connstring: 'averyrealdatabase'

Webhooks for config:

I'd like to implement a validating webhook for the config via a label. This should prevent a whole class of errors (like malformed yaml or missing stanzas) from affecting deployments.
Additionally, I'd like to use a mutating webhook to allow for integrating cluster-level resources where appropriate. This would probably work by looking at specially labeled ConfigMaps or Secrets, reading in one member as a config object, replacing parts with other various data, and then writing out the new config to another member. A quick theoretical example:

The webhook sees an object like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: clair-config
  labels:
    projectquay.io/clair/template-config: 'true'
  annotations:
    projectquay.io/clair/template-input: config.yaml.in
    projectquay.io/clair/template-output: config.yaml
data:
  config.yaml.in: |
    ---
    matcher:
      connstring: secret://database/DSN

It's selected for this behavior because of the label, and has annotations indicating what members of the object to look at.
The webhook would see the URI secret://database/DSN, resolve that to a key named DSN in a Secret named database, and substitute the value. (The URI usage not final, but seems promising.) The resulting object looks like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: clair-config
  labels:
    projectquay.io/clair/template-config: 'true'
  annotations:
    projectquay.io/clair/template-input: config.yaml.in
    projectquay.io/clair/template-output: config.yaml
data:
  config.yaml.in: |
    ---
    matcher:
      connstring: secret://database/DSN
  config.yaml: |
    ---
    matcher:
      connstring: 'database=clair user=user'

4 CRs

I think the best way to go about things is going to be to make 4 separate CRs: Indexer, Matcher, Notifier, and Clair.

The first 3 are mostly identical and will share a bunch of their logic; they'll run containers in their respective mode. They'll have a weak reference to a ConfigMap/Secret for their config, and then own HPA, Service, Deployment, and Monitoring objects.

The Clair object will be the 'batteries-included" entrypoint. It will own the configuration ConfigMap/Secret, a database Deployment if configured, an Ingress/Route, and the other CRs. The created ConfigMap/Secret will be opted-in to validating and templating, and use that machinery when the cluster changes.

The cluster capability detection should be able to be shared across all the CRs, and the per-service objects will probably share the bulk of their reconcile loop.

This means the CRs will look roughly like:

--
apiVersion: v1
kind: clair.projectquay.io/Indexer
metadata:
  name: indexer
  namespace: default
spec:
  config:
    namespace: default
    name: clair-config