Operation Monitoring

[jlashner]

This PR adds the agent_operations feed for the registry agent which allows for the monitoring of individual agent operations, and makes it possible to set alerts for failed processes in grafana.

Description

The agent heartbeat now publishes a dict of operation codes for the agent which can be used to determine the state of each operation that the agent can run. The operation codes are the following:

• NONE: If an operation has never been run
• STARTING: If an operation currently has the 'starting' status
• RUNNING: If the operation has the 'running' status
• STOPPING: If the operation has the 'stopping' status
• SUCCEEDED: If the operation has finished with success=True
• FAILED: If the operation has finished with success=False
• EXPIRED: If the agent who owns the operation is no longer running (set by the registry).

Motivation and Context

This lets you make cool grafana panels that track the status of all operations:

It also makes it possible to setup grafana alerts for individual operations instead of having to monitor docker updates with telegraf.

How Has This Been Tested?

This has been tested by running agents locally.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• Unless I am preparing a release, I have opened this PR onto the develop branch.

[BrianJKoopman]

This is great, I'm sure this functionality will see a lot of use. A couple of small suggestions inline, mostly to suggest documentation. The screenshot of the panel might even be good to include.

What'd you mean by "... instead of having to monitor docker updates with telegraf." Are you referring to the logs?

[BrianJKoopman]

This class will be autodoc'd and show up in the API section of the docs. It'd be good to link to that from the Registry page, as well as to update the registry documentation to describe the new op state tracking functionality. That'll make it easy for people to create a value mapping in Grafana.

[mhasself]

Cool!

This approval is contingent on Brian being happy with the documentation.

[jlashner]

Alright, I added a docs section and implemented all suggestions. I also added field name validation to the registry fields before adding them to the data block because I realized long or improper agent instance-id's would result in improper field names in the agent_operations feed.

Let me know if there's anything else I should add before merging!

[BrianJKoopman]

That's a good point about the field naming potentially becoming invalid. I commented on this inline below. I think moving this check closer to the heartbeat could provide the Agent user more feedback.

[jlashner]

Fixed Brian's last comments. Added Provider._enforce_feed_name_rules as a final step to make sure fields are monitored instead of dropped.

[BrianJKoopman]

Changes look good, thanks for those! Maybe longer term that method marked as hidden on the Provider should be moved somewhere more general or made not hidden, but this works for now.