Config syntax issues not bubbled up to user after check

[tmcarr]

Output of the commands on the info page:

root@freenas:/etc/datadog-agent/conf.d/process.d# sudo systemctl status datadog-agent
● datadog-agent.service - "Datadog Agent"
   Loaded: loaded (/lib/systemd/system/datadog-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-01-31 16:57:42 PST; 1h 8min ago
 Main PID: 4342 (agent)
   CGroup: /system.slice/datadog-agent.service
           ├─4342 /opt/datadog-agent/bin/agent/agent start -p /opt/datadog-agent/run/agent.pid
           ├─4384 /opt/datadog-agent/embedded/bin/process-agent
           └─4385 /opt/datadog-agent/embedded/bin/trace-agent

Jan 31 18:05:59 freenas.lan agent[4342]: 2018-01-31 18:05:59 PST | INFO | (datadog_agent.go:127 in LogMessage) | (transport.py:1572) | Authentication (password) successful!
Jan 31 18:05:59 freenas.lan agent[4342]: 2018-01-31 18:05:59 PST | INFO | (datadog_agent.go:127 in LogMessage) | (sftp.py:129) | [chan 0] Opened sftp connection (server version 3)
Jan 31 18:06:14 freenas.lan agent[4342]: 2018-01-31 18:06:14 PST | INFO | (datadog_agent.go:127 in LogMessage) | (connectionpool.py:805) | Starting new HTTPS connection (1): home.tmcarr.com
Jan 31 18:06:14 freenas.lan agent[4342]: 2018-01-31 18:06:14 PST | INFO | (datadog_agent.go:127 in LogMessage) | (transport.py:1572) | Connected (version 2.0, client OpenSSH_7.6p1)
Jan 31 18:06:14 freenas.lan agent[4342]: 2018-01-31 18:06:14 PST | INFO | (datadog_agent.go:127 in LogMessage) | (transport.py:1572) | Authentication (password) successful!
Jan 31 18:06:14 freenas.lan agent[4342]: 2018-01-31 18:06:14 PST | INFO | (datadog_agent.go:127 in LogMessage) | (sftp.py:129) | [chan 0] Opened sftp connection (server version 3)
Jan 31 18:06:29 freenas.lan agent[4342]: 2018-01-31 18:06:29 PST | INFO | (datadog_agent.go:127 in LogMessage) | (connectionpool.py:805) | Starting new HTTPS connection (1): home.tmcarr.com
Jan 31 18:06:29 freenas.lan agent[4342]: 2018-01-31 18:06:29 PST | INFO | (datadog_agent.go:127 in LogMessage) | (transport.py:1572) | Connected (version 2.0, client OpenSSH_7.6p1)
Jan 31 18:06:29 freenas.lan agent[4342]: 2018-01-31 18:06:29 PST | INFO | (datadog_agent.go:127 in LogMessage) | (transport.py:1572) | Authentication (password) successful!
Jan 31 18:06:29 freenas.lan agent[4342]: 2018-01-31 18:06:29 PST | INFO | (datadog_agent.go:127 in LogMessage) | (sftp.py:129) | [chan 0] Opened sftp connection (server version 3)

Datadog Version:

root@freenas:/etc/datadog-agent/conf.d/process.d# datadog-agent version
Agent 6.0.0-beta.9 - Commit: bf928b6 - Serialization version: 4.7

Additional environment details (Operating System, Cloud provider, etc):

Debian Sid

Steps to reproduce the issue:

1. Remove an opening bracket to make an invalid yaml file, for example:

  - name: ssh
    search_string: 'ssh', 'sshd']

2. Run datadog-agent check *

Describe the results you received:

root@freenas:/etc/datadog-agent/conf.d/process.d# datadog-agent check *
no check found
Error: no check found
Usage:
  agent check <check_name> [flags]

Flags:
  -r, --check-rate         check rates by running the check twice
  -d, --delay int          delay between running the check and grabbing the metrics in miliseconds (default 100)
  -h, --help               help for check
  -l, --log-level string   set the log level (default 'off')

Global Flags:
  -c, --cfgpath string   path to directory containing datadog.yaml
  -n, --no-color         disable color output

Describe the results you expected:

A check failed with the error:
/etc/datadog-agent/conf.d/process.d/conf.yaml is not a valid config file: yaml: line 28: did not find expected key

Additional information you deem important (e.g. issue happens only occasionally):

If you run datadog-agent with -l info, it clearly states the issue, but without the increased logging it leaves the user with no info as to why their configs were not checked.

See here:

root@freenas:/etc/datadog-agent/conf.d/process.d# datadog-agent check -l info *
2018-01-31 | INFO | (tagger.go:80 in Init) | starting the tagging system
2018-01-31 | INFO | (runner.go:93 in NewRunner) | Runner started with 4 workers.
2018-01-31 | INFO | (collector.go:51 in NewCollector) | Embedding Python 2.7.14 (default, Jan 25 2018, 16:49:29) [GCC 4.7.2]
2018-01-31 | INFO | (file.go:68 in Collect) | File Configuration Provider: searching for configuration files at: /etc/datadog-agent/conf.d
2018-01-31 | WARN | (file.go:179 in collectEntry) | /etc/datadog-agent/conf.d/process.d/conf.yaml is not a valid config file: yaml: line 28: did not find expected key
2018-01-31 | INFO | (file.go:68 in Collect) | File Configuration Provider: searching for configuration files at: /opt/datadog-agent/bin/agent/dist/conf.d
2018-01-31 | WARN | (file.go:72 in Collect) | Skipping, open /opt/datadog-agent/bin/agent/dist/conf.d: no such file or directory
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for apache at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for consul at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for couch at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for couchbase at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for elastic at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for etcd at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for kube_dns at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for kubernetes_state at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for kyototycoon at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for mcache at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for redisdb at this moment.
2018-01-31 | INFO | (autoconfig.go:289 in resolve) | Can't resolve the template for riak at this moment.
2018-01-31 | WARN | (check.go:243 in Configure) | could not get a check instance with the new api: __init__() takes at least 4 arguments (4 given)
2018-01-31 | WARN | (check.go:244 in Configure) | trying to instantiate the check with the old api, passing agentConfig to the constructor
2018-01-31 | WARN | (check.go:269 in Configure) | passing `agentConfig` to the constructor is deprecated, please use the `get_config` function from the 'datadog_agent' package (disk).
2018-01-31 | WARN | (check.go:243 in Configure) | could not get a check instance with the new api: __init__() takes at least 4 arguments (4 given)
2018-01-31 | WARN | (check.go:244 in Configure) | trying to instantiate the check with the old api, passing agentConfig to the constructor
2018-01-31 | WARN | (check.go:269 in Configure) | passing `agentConfig` to the constructor is deprecated, please use the `get_config` function from the 'datadog_agent' package (network).
no check found
Error: no check found
Usage:
  agent check <check_name> [flags]

Flags:
  -r, --check-rate         check rates by running the check twice
  -d, --delay int          delay between running the check and grabbing the metrics in miliseconds (default 100)
  -h, --help               help for check
  -l, --log-level string   set the log level (default 'off')

Global Flags:
  -c, --cfgpath string   path to directory containing datadog.yaml
  -n, --no-color         disable color output

[masci]

Thanks for the detailed report @tmcarr I was able to reproduce, we'll fix this shortly.

[mfpierre]

@tmcarr I improved the check command output, should be much clearer now to troubleshoot the issue, don't hesitate if you have additional feedback 👍