Removing entries from `datadog_checks` doesn't delete the file

[bgerstle]

Resulting in the datadog agent attempting to run checks which are likely fail.

Steps to reproduce

1. Configure playbook w/ datadog role & an arbitrary check
2. Run playbook
3. Remove entry from datadog_checks and replace it with a different one
4. Run playbook again

Expected result

Check is removed and agent is restarted, only running the new check.

Actual result

Old check is still run, as the /etc/dd-agent/checks.d/<check>.yaml file is still present.

Ideally we'd be doing immutable deploys and this wouldn't be an issue, but c'est la vie.

[olivielpeau]

Hi @bgerstle,
Thanks for your feedback! Indeed once a check config is not listed in datadog_checks the role stops keeping track of the config file, so it doesn't delete it.

There are a few different things that could be done by the role to address the case you described:

• have a variable (named datadog_disabled_checks for instance) that would allow listing checks for which the role would make sure no configuration file is present in /etc/dd-agent/checks.d/. This would still allow users to have check configs that are not managed by the role (as long as they're not listed in the variable, the role wouldn't touch them).
• a simpler but less flexible approach would be to have a flag variable (named datadog_disable_untracked_checks for instance) that would make the role remove all the check configs in /etc/dd-agent/checks.d/ that are not tracked.

Any thoughts?

[jharley]

I'm into the idea of something like datadog_dsiable_untracked_checks for hosts that aren't immutable.

[bgerstle]

@olivielpeau sorry for the (very) delayed response! Those options sound usable, but I'm curious what you think of either:

1. Following the apparent Ansible pattern of using a state variable which allows users to specify state: absent
2. Removing all checks except those specified for the given host (including default checks like ntp & such)

[jeffwidman]

I prefer the flexibility of being able to have checks that we manage outside of Ansible. For example, sometimes when I'm developing, I end up needing to sanity test checks in production and I don't want those checks removed if someone else unbeknownst to me runs the Ansible play.

[bgerstle]

Actually, giving this more thought I agree with @jeffwidman, but a more important use case for us is "stacking" datadog checks. We use Ansible to configure our hosts & services, as well as Datadog. So, in each service role, we can include the Datadog role as a dependency and define the necessary checks there. For example, our HAProxy role will define checks for HAProxy, and on that same machine, we'll have a Docker role which defines other checks to be configured alongside HAProxy.

To summarize:

• Don't remove other checks automatically to allow for unmanaged checks & "stacking" checks across different roles/plays
• But, still would be nice to be able to explicitly remove checks with a state: absent or similar flag.

[olivielpeau]

@bgerstle Thanks for the details, so if I understand correctly with an example:

datadog_checks:
  haproxy:
    init_config:
       [...]
    instances:
       [...]
  docker:
    state: absent
    init_config:
       [...]
    instances:
       [...]

You'd like these variables to enable the haproxy check but explicitly disable the docker check, correct?

[bgerstle]

@olivielpeau yeah, that should do it!

[mgu]

I find it simpler to remove the checks automatically
Using additional_checksd, unmanaged checks could be put somewhere else, right ?

[Jno21]

I made a proposal with my PR. Some feedback will be greatly appreciated :)

[rromanchuk]

can we merge #151 it should be no-op for anyone who doesn't need it, would prefer not to manage my own fork

[lesteenman]

This issue seems dead, but I can't find the feature yet. What's the current status?