Check datadog agent info for warnings & errors, failing the play if any are found

[bgerstle]

As a user, I expect an attempt to configure the datadog-agent improperly would result in a play failure. Instead, it fails silently and I don't realize monitoring is broken until after the fact.

It seems this could be accomplished with something along these lines:

- name: Get datadog-agent status
  command: /etc/init.d/datadog-agent info
  register: agent_info

- fail: "Something seems wrong with your configuration, please check the docs"
  when: 'WARNING' in agent_info or 'ERROR' in agent-info

[jeffwidman]

👍 for this idea, although it may need some tweaking.

For example, we run a custom proxy between our agents and Datadog, so the agents always throw a warning for us: DataDog/dd-agent#3346 (comment)

Not sure if the when statement only matches on casing/exact word or partial words...

[bkabrda]

Hi, so right now, the restart datadog-agent handler will fail if the agent fails to start - I just tested with a malformed config and the whole play failed because of the handler failing. The general problem with things like this is how long do you want to wait to see errors/warnings? The agent may fail 1 second after we check, so we can never get this 100 % right, even if we waited for a long time (which would make the play very long).

I believe the current state of the role is sufficient in this sense - if you have an example of misconfiguration that made the agent fail while the play finished fine, feel free to open another issue with more specifics, maybe there's a different way to solve that case.

Thanks!