Add ability to watch a key in Consul and trigger jobs based on changes to the value for that key

[misterbisson]

I need to be able to make changes to a configuration template outside of the Docker image. I'd typically argue that the config file is code and needs to be versioned with code, but that's not exactly the same as saying it needs to be deployed as an update to the image.

So, I'm looking for a way to keep the config template in Consul (in the k/v store) and have ContainerPilot watch watch the key for any changes, then trigger a job if it detects changes. I can write a job that will fetch that config template and poke my app with the update when it's complete, but I'd prefer to use the watch mechanism in ContainerPilot to do so.

[jwreagor]

I think this is another reasonable solution to use cases similar to #504. It's also nice that it doesn't rely on UNIX signal processing which can be a headache at times.

[tgross]

I think this is another reasonable solution to use cases similar to #504.

It's also useful in much more general cases. IIRC this was in discussion when we made the switch to Consul-only anyways. So 👍 on this feature.

[misterbisson]

Two issues that I can think of are that the CP config for watches assumes only services, so we'll have to figure out a path through that. I also can't remember if CP is now updated to use true Consul watches or if it's still polling (which was definitely something we had in mind in the switch to Consul only).

[tgross]

Two issues that I can think of are that the CP config for watches assumes only services, so we'll have to figure out a path through that.

I'm pretty sure @geek had a good proposed config for that, it just never made it into an issue and I can't remember it (or go back through Slack now! 😀 )

I also can't remember if CP is now updated to use true Consul watches or if it's still polling (which was definitely something we had in mind in the switch to Consul only).

Not yet. See #461

[jwreagor]

Starting to sketch out the order of operations for this task. Looks something like...

• Expand upon the definition that watches are only tied to services (remove serviceName from CP's Watch type)
• Add a new watchType field to watches that pairs with the existing Consul Watch API.
• If not specified otherwise, default watchType will be service for backwards compat.
• Implement Blocking Queries for the existing Discovery backend.
• Implement a handful of new Event types or enhance what we have for watching keys, keyprefix, nodes (?), etc.
• ???
• Profit

I'll flesh this out a bit more in the coming days.

[Smithx10]

This is an excellent idea. @cheapRoc That seems like a profitable avenue to go down.