Custom file logging not working when logs come from a non integration source file

[adragoset]

https://docs.datadoghq.com/logs/#custom-log-collection

Under the custom file section the user is instructed to create a config.yaml and place it in

custom-integration-name.d/conf.yaml

init_config:

instances:
    [{}]

logs:
  - type: file
    path: some/file/path
    service: custom-integration-name
    source: custom-integration-name

However folders placed into conf.d and mounted do not get copied into /etc/datadog-agent/conf.d/

If the user names the config file custom-integration-name.yaml and places it into /conf.d thats mounted datadog then throws an error

Datadog's nomad_forwarder integration is reporting:
Instance #initialization[ERROR]:{"Core Check Loader":"Check custom-integration-name not found in Catalog","JMX Check Loader":"check is not a jmx check, or unable to determine if it's so","Python Check Loader":"No module named custom-integration-name"}

If the user is not using an existing integration and just wants to tail the file it does not look like this is possible.

[adragoset]

closing due to config error.

[NBParis]

Hello @adragoset ,

A bit too fast in the closing so was not able to answer before :)
However we will update the documentation to remove the init_config and instances from the custom file examples.

As indeed, if you are using a yaml file only for logs, you do not need those sections and using them will raise this kind of errors.
However, those can be safely ignored.
We will make sure to update the documentation so the behaviour is clearly explained.

[adragoset]

Hi thank you for answering ill remove those sections and see if it resolves errors showing up in the logs.

[adragoset]

@NBParis

I was able to make the above errors go away however it looks like the agent is failing to connect to the datadog log endpoint.

Here is an excerpt from the log output

[ AGENT ] 2018-03-16 16:41:54 UTC | INFO | (connection_manager.go:58 in NewConnection) | Connecting to the backend: intake.logs.datadoghq.com:10516
[ AGENT ] 2018-03-16 16:41:54 UTC | WARN | (docker.go:199 in forwardMessages) | Can't parse docker message: expected a whitespace after header
[ AGENT ] 2018-03-16 16:41:54 UTC | WARN | (docker.go:199 in forwardMessages) | Can't parse docker message: expected a whitespace after header
[ AGENT ] 2018-03-16 16:41:54 UTC | WARN | (docker.go:199 in forwardMessages) | Can't parse docker message: expected a whitespace after header
[ AGENT ] 2018-03-16 16:41:54 UTC | WARN | (docker.go:199 in forwardMessages) | Can't parse docker message: expected a whitespace after header
[ AGENT ] 2018-03-16 16:41:54 UTC | WARN | (docker.go:199 in forwardMessages) | Can't parse docker message: expected a whitespace after header
[ AGENT ] 2018-03-16 16:41:54 UTC | INFO | (tagger.go:77 in Init) | starting the tagging system
[ AGENT ] 2018-03-16 16:41:54 UTC | INFO | (tagger.go:151 in tryCollectors) | docker tag collector successfully started
[ AGENT ] 2018-03-16 16:41:54 UTC | WARN | (connection_manager.go:65 in NewConnection) | dial tcp 35.173.65.77:10516: getsockopt: connection refused

As you can see im getting connection refused when it tries to ship logs to that endpoint. I am behind a proxy with a self signed cert could that be causing the issue?

[NBParis]

@adragoset Indeed that would be an issue as the agent is targeting intake.logs.datadoghq.com on the port 10516.

I would suggest to open a support ticket with your proxy setup to support@datadoghq.com so we can investigate the issue.
Sending a flare of the agent would also be very helpful (https://github.com/DataDog/datadog-agent/blob/master/docs/agent/changes.md#linux).
Thanks,
Nils