Run Clair with k8s

[odidev]

Hi Team,

I am working with Clair using the latest v4.4.4.

I successfully ran the Combo and Distributive modes of Clair using docker-compose.yaml.

Now, I am working on running the distributive mode on the k8s cluster. For this, I will be using minikube.

I found this old commit here, and it seems that the helm method of deployment has been removed from the source code and the documentation.

Also, I could not find any config in the source code that can be used to deploy Clair on k8s.

May I know, do you support running Clair with k8s? And if yes, can you please point me to the official Clair’s documentation for the same?

[hdonnay]

You certainly can run Clair distributed in kubernetes. The setup is discussed in broad strokes here. As that details, you'll need at least one database engine, a load balancer/router, and 3 deployments; the deployments should share the config, but each have the CLAIR_MODE (or mode flag) provided appropriately.

[crozzy]

There is also this manifest that we use for deploying Clair as a service (disclaimer: it's openshift specific so there's probably some k8s translation to do and there are some specific parts that might only be useful when operating at large-scale, i.e. the indexer volumes).

[odidev]

Thank you for the suggestions @hdonnay and @crozzy

Following the docker-compose.yaml file, I tried creating kubernetes configurations for all 3 modes and the load balancer.
I created 3 pods, each having a container for postgres database and another container for the mode of clair, I.e. indexer, matcher and notifier.

For example, below is the config.yaml for indexer:

apiVersion: v1 
kind: Pod 
metadata: 
  name: distributive-indexer 
  labels: 
    apps: distributive-indexer 
spec: 
  containers: 
    - name: clair-database 
      image: postgres:12 
      env: 
        - name: POSTGRES_HOST_AUTH_METHOD 
          value: "trust" 
      ports: 
        - containerPort: 5432 
      volumeMounts: 
        - name: db-details 
          mountPath: /docker-entrypoint-initdb.d/init.sql 
- name: clair-indexer 
      image: quay.io/projectquay/clair:4.4.4 
      volumeMounts: 
        - name: config-file 
          mountPath: /tmp/config.yaml 
      env: 
        - name: CLAIR_MODE 
          value: "indexer" 
        - name: CLAIR_CONF 
          value: "/tmp/config.yaml" 
  volumes: 
    - name: db-details 
      hostPath: 
        path: /minikube-host/CLAIR/clair-v4.4.4/local-dev/clair/init.sql 
        type: File 
    - name: config-file 
      hostPath: 
        path: /minikube-host/CLAIR/clair-v4.4.4/local-dev/clair/config.yaml 
        type: File 

Similarly I created 2 more config files for matcher and notifier resp and a load balancer. Kindly find them below:

1. indexer: clair-indexer.txt
2. matcher: clair-matcher.txt
3. notifier: clair-notifier.txt
4. traefik: clair-traefik.txt

I can see that all 3 pods are running and the matcher logs show that it is filling the vulnerabilities in it’s database. However, notifier logs sometimes show that it is failing to lookup for clair-matcher.

Next, I tried creating another pod with a container that runs traefik as a load balancer. Below is the configuration:

apiVersion: v1 
kind: Pod 
metadata: 
  name: distributive-traefik 
  labels: 
    apps: distributive-traefik 
spec: 
  containers: 
    - name: clair-traefik 
      image: traefik:v2.2 
      volumeMounts: 
        - name: traefik-config 
          mountPath: /etc/traefik/ 
      ports: 
        - containerPort: 6060 
        - containerPort: 8080 
        - containerPort: 8443 
        - containerPort: 5432 
  volumes: 
    - name: traefik-config 
      hostPath: 
        path: /minikube-host/CLAIR/clair-v4.4.4/local-dev/traefik/ 
        type: Directory 

But the traefik logs show as below:

time="2022-09-01T11:41:51Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yaml" 
time="2022-09-01T11:41:51Z" level=error msg="server not found" 
time="2022-09-01T11:41:52Z" level=error msg="In service \"postgresql@file\" server \"clair-database:5432\": lookup clair-database on 10.96.0.10:53: server misbehaving" routerName=postgresql@file entryPointName=postgresql serviceName=postgresql 
time="2022-09-01T11:41:52Z" level=error msg="server not found" 
time="2022-09-01T11:41:52Z" level=error msg="In service \"postgresql@file\" server \"clair-database:5432\": lookup clair-database on 10.96.0.10:53: server misbehaving" serviceName=postgresql entryPointName=postgresql routerName=postgresql@file 

It seems traefik is looking for database, but my understanding is that traefik will only load balance all 3 modes of clair, and it does not require database. I even tried creating a database container in traefik pod, but it didn’t help.

May I know is this the correct configuration that I use? If not, can you please suggest me what else can be changed to load balance all 3 modes of clair?

FYI, I am running clair on minikube.

[hdonnay]

Are you running that inside your cluster? You need to expose the ingress somehow, if not.

[odidev]

Yes, following this document < https://platform9.com/blog/building-a-complete-stack-ingress-controllers/ >, I have already exposed the ingress controller. Below are the concerned logs:

$ kubectl get services ingress-nginx-controller --namespace=ingress-nginx 

NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE 
ingress-nginx-controller   LoadBalancer   172.20.36.57   <pending>     80:32628/TCP,443:30044/TCP   27m 

Then I submitted the manifest at http://172.20.36.57:32628, and it returned timeout issue.
Submitting the manifest at port 80 also didn’t help.

I tried replacing the port name “clair-http” with port number “80” in the ingress configuration in clair’s config file, but that didn’t help.

[hdonnay]

I don't really have the expertise to debug an ingress, but they usually have an "external IP". Cluster IPs only work within the cluster, I do know that.

[odidev]

I used terraform to create AKS cluster, and then deployed ingress and clair. I am able to get external IP for ingress controller.

I submitted the manifest on External_IP at port 80, and got error 401 this time:

$ ./clairctl-linux-arm64 -D report --host "[http://20.85.45.210:80](http://20.85.45.210/)" ubuntu 

2023-01-11T11:36:59Z DBG fetching ref=ubuntu 
2023-01-11T11:36:59Z DBG using text output 
2023-01-11T11:36:59Z DBG found manifest digest=sha256:965fbcae990b0467ed5657caceaec165018ef44a4d2d46c7cdea80a9dff0d1ea ref=ubuntu 
2023-01-11T11:36:59Z DBG requesting index_report attempt=1 digest=sha256:965fbcae990b0467ed5657caceaec165018ef44a4d2d46c7cdea80a9dff0d1ea ref=ubuntu 
2023-01-11T11:36:59Z DBG digest=sha256:965fbcae990b0467ed5657caceaec165018ef44a4d2d46c7cdea80a9dff0d1ea method=GET path=/indexer/api/v1/index_report/sha256:965fbcae990b0467ed5657caceaec165018ef44a4d2d46c7cdea80a9dff0d1ea ref=ubuntu status="401 Unauthorized" 
2023-01-11T11:36:59Z DBG index error error="unexpected return status: 401" digest=sha256:965fbcae990b0467ed5657caceaec165018ef44a4d2d46c7cdea80a9dff0d1ea ref=ubuntu 
2023-01-11T11:36:59Z ERR error="unexpected return status: 401" 

It seems like the indexer received the manifest, but failed with the below logs:

$ kubectl logs pod/clair-app-indexer-bf8dc9c5d-fkw57 -n crozzy-cluster-clair 

{"level":"debug","component":"initialize/Logging","time":"2023-01-11T11:24:27Z","message":"logging initialized"} 
{"level":"info","component":"main","version":"v4.5.0-27-gadbaa567 (claircore v1.4.14-0.20230110180439-af604ea8f0c4)","time":"2023-01-11T11:24:27Z","message":"starting"} 
{"level":"info","component":"main","lint":"introspection address not provided, default will be used (at $.introspection_addr)","time":"2023-01-11T11:24:27Z"} 
{"level":"debug","component":"main","time":"2023-01-11T11:24:27Z","message":"found cgroups v2"} 
{"level":"info","component":"main","time":"2023-01-11T11:24:27Z","message":"no CPU quota set, using default"} 
{"level":"info","component":"main","cur":0,"prev":2,"time":"2023-01-11T11:24:27Z","message":"set GOMAXPROCS value"} 
{"level":"info","component":"main","version":"v4.5.0-27-gadbaa567 (claircore v1.4.14-0.20230110180439-af604ea8f0c4)","time":"2023-01-11T11:24:27Z","message":"ready"} 
{"level":"info","component":"main","time":"2023-01-11T11:24:27Z","message":"launching introspection server"} 
{"level":"info","component":"introspection/New","address":":8089","time":"2023-01-11T11:24:27Z","message":"no introspection address provided; using default"} 
{"level":"warn","component":"introspection/New","time":"2023-01-11T11:24:27Z","message":"no health check configured; unconditionally reporting OK"} 
{"level":"info","component":"introspection/Server.withPrometheus","endpoint":"/metrics","server":":8089","time":"2023-01-11T11:24:27Z","message":"configuring prometheus"} 
{"level":"info","component":"introspection/New","time":"2023-01-11T11:24:27Z","message":"no distributed tracing enabled"} 
{"level":"info","component":"main","time":"2023-01-11T11:24:27Z","message":"launching http transport"} 
{"level":"info","component":"initialize/Services","time":"2023-01-11T11:24:27Z","message":"begin service initialization"} 
{"level":"info","component":"main","time":"2023-01-11T11:24:27Z","message":"registered signal handler"} 
{"level":"debug","component":"internal/ctxlock/Locker.reconnect","gen":"1","time":"2023-01-11T11:24:27Z","message":"set up"} 
{"level":"info","component":"libindex/New","time":"2023-01-11T11:24:27Z","message":"registered configured scanners"} 
{"level":"info","component":"initialize/Services","time":"2023-01-11T11:24:27Z","message":"end service initialization"} 
{"level":"info","component":"httptransport/New","path":"/openapi/v1","time":"2023-01-11T11:24:27Z","message":"openapi discovery configured"} 
{"level":"debug","component":"middleware/auth/PSK.Check","time":"2023-01-11T11:28:12Z","message":"failed to retrieve jwt from header"} 
{"level":"debug","component":"middleware/auth/PSK.Check","time":"2023-01-11T11:28:39Z","message":"failed to retrieve jwt from header"} 
{"level":"debug","component":"middleware/auth/PSK.Check","time":"2023-01-11T11:29:10Z","message":"failed to retrieve jwt from header"} 
{"level":"debug","component":"middleware/auth/PSK.Check","time":"2023-01-11T11:29:52Z","message":"failed to retrieve jwt from header"} 
{"level":"debug","component":"middleware/auth/PSK.Check","time":"2023-01-11T11:36:59Z","message":"failed to retrieve jwt from header"} 

I found a similar issue ticket in Clair < #1569 (comment) >.

As suggested in the issue, the resolution was to use same configuration file with clairctl, as is used in the authentication.

I created a configuration file with the same stringData as mentioned in the kubernetes config file you shared, and executed clairctl with --config=<config.yaml>, but this hasn’t solved the issue, rather I got error 404.

Do you have any suggestions here?

[hdonnay]

Make sure the intraservice addresses are correct is my only suggestion.

[eldoranstars]

Hi Team! I am new at Clair, but try to dive deeper =)
I am working with Clair using the https://github.com/quay/clair/tree/release-4.4
I try to run the distributive mode on the k8s cluster. For this, I using OpenShift version 4.11.0-0.okd-2022.
I found this template https://github.com/quay/clair/blob/release-4.4/contrib/openshift/manifests/manifests.yaml
I just replace "port: 80" to "port: 8080" in Services and delete Probes from deployments..
I took configuration from here https://github.com/quay/clair/blob/release-4.4/config.yaml.sample

show configuration

introspection_addr: localhost:8089
http_listen_addr: localhost:8080
log_level: debug
indexer:
  connstring: host=postgresql port=5432 user=clair password=clair dbname=clair sslmode=disable
  scanlock_retry: 10
  layer_scan_concurrency: 5
  migrations: true
matcher:
  indexer_addr: http://clair-indexer:8080/
  connstring: host=postgresql port=5432 user=clair password=clair dbname=clair sslmode=disable
  max_conn_pool: 100
  run: ""
  migrations: true
  updater_sets:
  - "alpine"
  - "aws"
  - "debian"
  - "oracle"
  - "photon"
  - "pyupio"
  - "rhel"
  - "suse"
  - "ubuntu"
matchers:
  names:
  - "alpine"
  - "aws"
  - "debian"
  - "oracle"
  - "photon"
  - "python"
  - "rhel"
  - "suse"
  - "ubuntu"
  - "crda"
  config:
    crda:
      url: https://gw.api.openshift.io/api/v2/
      source: clair-sample-instance
      key: 207c527cfc2a6b8dcf4fa43ad7a976da
notifier:
  indexer_addr: http://clair-indexer:8080/
  matcher_addr: http://clair-matcher:8080/
  connstring: host=postgresql port=5432 user=clair password=clair dbname=clair sslmode=disable
  migrations: true
  delivery_interval: 1m
  poll_interval: 5m
  # if multiple delivery methods are defined the only one will be selected.
  # preference order:
  # webhook, amqp, stomp
  webhook:
    target: "http://webhook/"
    callback: "http://clair-notifier/notifier/api/v1/notifications"
  amqp:
    exchange:
        name: ""
        type: "direct"
        durable: true
        auto_delete: false
    uris: ["amqp://user:pass@host:10000/vhost"]
    direct: false
    routing_key: "notifications"
    callback: "http://clair-notifier/notifier/api/v1/notifications"
    tls:
    root_ca: "optional/path/to/rootca"
    cert: "madatory/path/to/cert"
    key: "madatory/path/to/key"
  stomp:
    desitnation: "notifications"
    uris: ["stomp:10000/vhost"]
    direct: false
    callback: "http://clair-notifier/notifier/api/v1/notifications"
    login:
      login: "username"
      passcode: "passcode"
    tls:
    root_ca: "optional/path/to/rootca"
    cert: "madatory/path/to/cert"
    key: "madatory/path/to/key"
trace:
  name: "jaeger"
  probability: 1
  jaeger:
    agent:
      endpoint: "localhost:6831"
    service_name: "clair"
metrics:
  name: "prometheus"

My deploy in cluster look like

I get 404 when I try to curl 8080. What's wrong?

clair-index terminal

sh-4.4$ curl localhost:8080

404 page not found

sh-4.4$ curl localhost:8089

404 page not found

sh-4.4$ curl localhost:8089/healthz

ok

As I see in documentation, clair services works on 6060 https://quay.github.io/clair/howto/testing.html#starting-a-cluster
Why 6060 is absent in https://github.com/quay/clair/blob/release-4.4/contrib/openshift/manifests/manifests.yaml ?

[hdonnay]

• You need to request an endpoint that exists.
• The equivalent is at the bottom, HTTP_TRANSPORT_PORT.

[eldoranstars]

• You need to request an endpoint that exists.
• The equivalent is at the bottom, HTTP_TRANSPORT_PORT.

• but endpoint give me 404, why? https://github.com/quay/clair/blob/release-4.4/contrib/openshift/manifests/manifests.yaml#L367
• i need port 8080 or 6060 for Clair services on localhost? https://quay.github.io/clair/howto/testing.html#starting-a-cluster

[hdonnay]

• You didn't request an endpoint that exists, based on the pasted output. When you did (in the last instance) it replied with a 200.
• It looks like you have it configured to listen on 8080.

[eldoranstars]

• You didn't request an endpoint that exists, based on the pasted output. When you did (in the last instance) it replied with a 200.
• It looks like you have it configured to listen on 8080.

TY! It works now. But i need to connect my private repo. How can i do that? Can you help me plz?