Distributed Deployment - ALB Healthchecks

[mieliespoor]

We looking at deploying clair in a distributed manner where we will have three different containers (ECS Services) running for the indexer, matcher and notifier services as outlined in the howto.

An AWS ALB makes for very easy path based routing, but each ALB TargetGroup then also need a healthcheck endpoint. For the indexer, this can be somewhat faked by using /indexer/api/v1/index_state and then checking for the http status code being 200.

There is however no endpoint I can see that can be used for the notifier and matcher, or is there some hidden /health path that I'm missing?

My understanding is that v2 provided for a health check path on port 6061, but its not clear if it is the case on v4 here?

[mieliespoor]

So when looking at main.go ln 86, I see this:

i, err := introspection.New(srvctx, conf, nil)

This means that I won't ever see the health checks working using the default code. This I would say is less than ideal. Without me changing any code, how would we get this working?

Ideally I don't want to change code as we don't want to compile and maintain this ourselves. We only want to use it 'off the shelf', so ideally only want to pull docker images to use Clair.

[hdonnay]

There's a health endpoint on the introspection port, which runs by default. The configuration should be in the documentation and the address used is printed in the logs.

[mieliespoor]

looking at the logs for the notifier (the matcher and indexer print much the same logs), I see this:

11:53PM DBG logging initialized component=initialize/Logging
11:53PM INF starting component=main version=v4.3.5
11:53PM INF ready component=main version=v4.3.5
11:53PM INF launching introspection server component=main
11:53PM INF launching http transport component=main
11:53PM INF registered signal handler component=main
11:53PM WRN no health check configured; unconditionally reporting OK component=introspection/New
11:53PM INF configuring prometheus component=introspection/Server.withPrometheus endpoint=/metrics server=:8089
11:53PM INF configuring jaeger exporter to push to agent component=introspection/Server.withJaeger endpoint=jaeger:6831
11:53PM INF begin service initialization component=initialize/Services
11:53PM INF performing notifier migrations component=notifier/service/storeInit
11:53PM INF initializing notifier store component=notifier/service/storeInit
11:53PM INF initializing poller component=notifier/service/New interval=15s
11:53PM INF initializing processors component=notifier/service/New count=4

My config:

log_level: debug-color
introspection_addr: ":8089"
http_listen_addr: ":6000"

Not sure where the healthcheck then would be?

Also not sure which documentation you refer to as the documentation on clair is pretty sparse. This here has no reference to any healthchecks.

[hdonnay]

yeah, you've got it configured for port 8089, and the endpoint is /healthz

[mieliespoor]

OK that is great. So we only need to call the health endpoint on /healthz not the path of the service also.

This help a lot, thanks. We will now be able to deploy the individual services and be able to scale them separately if required. We will then try this out tomorrow during our day.

@cvmcalister ^^

[hdonnay]

Great! I'll make a note to try to flesh out the documentation, when I get a chance.

[mieliespoor]

Just to reply in a separate message to be able to mark this question as answered.

To enable health checks on the container, you need to check the /healthz endpoint on the introspection port. The default configuration has introspection running on port :8089, so the health check would then be http://localhost:8089/healthz

Keep in mind that this port might not be exposed by the container, so in addition to the service port, default being :6060 port :8089 might also need to be exposed.

When you then run you container in something like AWS ECS, the ALB configuration would look something like this...
ALB:
Listener:
Port: 443:6060
TargetGroup:
Port: 6060
HealtCheck Port: 8089
HealthCheck Path: /healthz

The above configuration will then only expose the service on port 443 and port 8089 only available to the ALB. The ALB will handle TLS offloading and allow for unencrypted traffic into the container.

I hope this will help someone else.