APM Trace agent doesn't use http proxy configuration for reporting

[jeremyolliver]

I'm attempting to use APM in an environment that restricts outbound http unless through an http proxy, and from what I can tell this agent is not supporting the http proxy config (or I'm missing the correct place to put it).

I do have at present (proxy_host, proxy_port, proxy_user, proxy_password) set inside /etc/dd-agent/datadog.conf and have restarted the datadog-agent.

I see the following in my logs:

tail -f /var/log/datadog/trace-agent.log
2017-03-16 14:21:50 ERROR (endpoint.go:119) - error when requesting to endpoint https://trace.agent.datadoghq.com/api/v0.1/collector: Post https://trace.agent.datadoghq.com/api/v0.1/collector?api_key=[redacted]: dial tcp 23.23.189.219:443: i/o timeout
2017-03-16 14:21:52 INFO (receiver.go:329) - receiver handled 51 spans, dropped 0 ; handled 51 traces, dropped 0
2017-03-16 14:22:06 ERROR (endpoint.go:119) - error when requesting to endpoint https://trace.agent.datadoghq.com/api/v0.1/collector: Post https://trace.agent.datadoghq.com/api/v0.1/collector?api_key=[redacted]: dial tcp 50.16.188.192:443: i/o timeout

[talwai]

Hi @jeremyolliver . The trace agent does not support reporting through a proxy in the way that the infra agent does, so your settings in /etc/dd-agent/datadog.conf are not being respected for traces. It's something we are targeting for an upcoming release. In the meantime could you tell us more about your proxy setup - are you using something like HAProxy / squid or just proxying through other datadog-agents ?

[jeremyolliver]

No - the application servers I'm monitoring are not permitted outbound on port 80, and 443 (hence the i/o timeout error - this network access is intentionally blocked). To make any outbound http calls at all, they are required to connect through an internal http proxy server, which itself will forward on the http(s) request. I'm not having any issues with the datadog-agent reporting host and integration metrics - but as mentioned the trace-agent appears to send it's own metrics rather than pass them through the same mechanism used there.

[ufoot]

AFAIK we do not override the standard Go net/http client so standard overrides should work (https://golang.org/src/net/http/transport.go?s=9778:9835#L535 for example, it's about setting the env var HTTP_PROXY). As @talwai said, indeed, we should use the proper /etc/dd-agent/datadog.conf version in further releases.

[talwai]

Yea echoing @ufoot 's point, @jeremyolliver is it feasible for you to set the following env vars on your hosts running the trace-agent?

HTTP_PROXY=http://USERNAME:PASSWORD@PROXYIP:PROXYPORT
HTTPS_PROXY=https://USERNAME:PASSWORD@PROXYIP:PROXYPORT

this should workaround the problem while we address the underlying issue of not reading proxy settings from datadog.conf

[talwai]

fixed in PR #251 and will be released very shortly

[whithajess]

@talwai It seems the proxy environment variables are set and I can even see them under the dd-agent user is there possibly some kind of other problem causing the error @jeremyolliver reported?

[talwai]

hmm @whithajess that's unexpected - i'd expect the env vars to do the trick, if they're indeed available to the running process. is your error identical to the one above? can you paste a snippet of /var/log/datadog/trace-agent.log that shows the failure?

[whithajess]

Yes sorry for clarification I work with Jeremy so it is identical to above.

[whithajess]

I wrote a test go script using net/http and it does follow https_proxy is there some kind of other communication the data dog agent does? @talwai

[talwai]

Hi @whithajess @jeremyolliver thanks for working through this. datadog-agent version 5.12.1 has the patch you need for correct proxy configuration. More info here