Telemetry health checks

[adamlc]

Its great to see theres now telemetry. I think a couple of small updates need to be made. I see that the telemetry service registers a health check in consul, which is fine but it doesn't seem like the name is configurable and it seems the healthcheck doesn't get deleted when the container stops.

Should hopefully be changed easily enough :)

[tgross]

it seems the healthcheck doesn't get deleted when the container stops

That's definitely a bug... I'll see what I can do about tracking it down.

[misterbisson]

it doesn't seem like the [telemetry service] name is configurable

True. The telemetry service name is not configurable, as it represents ContainerPilot itself. The roadmap for the telemetry service includes presenting additional metadata about the container. Predictability in the service name is intended as a way to simplify the development of consumers and new thinking about how this data can be used.

The service ​_does_​ support filtering using tags. You can add tags to the ContainerPilot config and look for them using the consul_sd_config options, even though that's not demonstrated in https://github.com/autopilotpattern/prometheus.

the healthcheck doesn't get deleted when the container stops

We agree, that's a bug we need to fix.

[adamlc]

The telemetry service name is not configurable, as it represents ContainerPilot itself. The roadmap for the telemetry service includes presenting additional metadata about the container.

That makes sense, I guess a consumer would just query all of the ContainerPilot services 😸

[tgross]

I need to get back to this one so I'm going to take a look at it this morning. We did a big refactoring of the config parsing since this issue was opened so the problem may have changed/fixed because of the way we're creating a service for the telemetry endpoint.

[tgross]

I've opened #155 with an integration test exercising the registration and deregistration of the telemetry endpoint in Consul. I can't reproduce this bug with this test or with "by hand" tests, at least with the latest version of ContainerPilot.

Can you confirm this is still a problem @adamlc and maybe provide a Compose file and ContainerPilot config that reproduces the issue if so?

[adamlc]

I'll take another look in the next few days @tgross 👍

[justenwalker]

@adamlc One thing I've seen that makes de-registration break is if you are using a consul cluster and registering/deregistering by DNS name that contains more than one entry (ie, one for each server in your cluster: consul.company.com:8500 ). It is best if you register your service against a local agent that is joined to the consul cluster as a client, rather than the consul servers directly. This is a problem regardless of being a telemetry service, or a container service.

[adamlc]

Sorry about the late reply guys, I've not been able to look at this yet! To be honest I was messing about locally so it could well be what @justenwalker said :)

[misterbisson]

@adamlc I'm going to close this one, but please re-open if you see the problem again.