Possible to use a proxy?

[2ZZ]

Hi,

Is it possible to deploy this app with an outbound proxy where direct access is not permitted?

2020-02-17T15:39:39.013691Z     error   newrelic        {"err":"error posting data: Post https://metric-api.newrelic.com/metric/v1: context deadline exceeded"}
2020-02-17T15:39:39.013731Z     error   newrelic        {"context-error":"context deadline exceeded","event":"harvest cancelled or timed out","message":"dropping data"}

I tried to pass the proxy as an environment variable

spec:
  template:
    spec:
      containers:
        - name: newrelic-istio-adapter
          image: "newrelic/newrelic-istio-adapter:2.0.1"
          env:
            - name: https_proxy
              value: http://myproxyurl:3128

But the pod never becomes ready

k -n newrelic-istio-adapter logs -f newrelic-istio-adapter-69f768449b-d8lq5
2020-02-17T15:15:55.501444Z     info    newrelic        {"api-key":"########","event":"harvester created","harvest-period-seconds":5,"metrics-url-override":"","spans-url-override":"","version":"0.1.0"}
2020-02-17T15:15:55.501620Z     info    newrelic        listening on "[::]:55912"
2020-02-17T15:15:55.501638Z     info    newrelic        built metrics: map[string]metric.info{}
2020-02-17T15:16:33.194348Z     info    newrelic        received SIGTERM, exiting gracefully...

[MrAlias]

@2ZZ : The adapter currently doesn't support configuring an HTTP(S) proxy for outgoing connections.

However, it does support overriding the two endpoints that it sends data to (New Relic API endpoints):

go run ./cmd/main.go --help
usage: main [<flags>] <api-key>

Flags:
...
      --metrics-host=METRICS-HOST       Endpoint to send metrics (used for debugging)
      --spans-host=SPANS-HOST           Endpoint to send spans (used for debugging)
...

Not sure if that helps as your proxy would have to be configured to forward those requests to the New Relic endpoints without the initial request being to those URL.

Let me know if that doesn't help. I can see about adding proxy support if not.

[2ZZ]

Hi @MrAlias, thanks, I can probably get by doing that while testing but if possible a proxy parameter would be much preferred.

[MrAlias]

@2ZZ : did you by chance try setting the http_proxy (instead of https_proxy)? From my understanding that is a natively supported Go environment variable (the https_proxy being supported by the ProxyFromEnvironment function, which is something we would have to integrate).

[2ZZ]

I did yes, I set http_proxy, https_proxy and the same in upper case.
It appeared to make a difference but for some reason the adapter pod exited with a SIGTERM
Possibly there is some traffic that needs to go into a no_proxy variable, but I am not sure what.

[MrAlias]

Interesting. I'm going to see if I can reproduce locally to dig further into this.

[MrAlias]

@2ZZ : I was able to reproduce the SIGTERM locally with your env settings. Looks like the healthcheck is failing based on the generic endpoint it is configured to use:

$ kubectl -n newrelic-istio-adapter describe po newrelic-istio-adapter-....
...
  Warning  Unhealthy  46s (x7 over 2m6s)   kubelet, gke-istio-debug-default-pool-68be8222-dnfc  Readiness probe failed: timeout: failed to connect service ":55912" within 1s
  Warning  Unhealthy  38s (x7 over 118s)   kubelet, gke-istio-debug-default-pool-68be8222-dnfc  Liveness probe failed: timeout: failed to connect service ":55912" within 1s

Using the following config got around the SIGTERM:

# Source: newrelic-istio-adapter/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: newrelic-istio-adapter
  namespace: newrelic-istio-adapter
  labels:
    ...
spec:
  ...
  template:
    ...
    spec:
      containers:
        - name: newrelic-istio-adapter
          image: "newrelic/newrelic-istio-adapter:2.0.1"
          ports:
            - name: grpc
              containerPort: 55912
          env:
            # not sure this is needed (pretty sure by default Go ignores this endpoint).
            - name: no_proxy
              value: localhost
            - name: https_proxy
              value: http://myproxyurl:3128
            - name: NEW_RELIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: newrelic-istio-adapter
                  key: NEW_RELIC_API_KEY
          args:
            - --log-level
            - debug
            - --cluster-name
            - istio-cluster
            - $(NEW_RELIC_API_KEY)
          readinessProbe:
            exec:
              command: ["/bin/grpc_health_probe", "-addr=localhost:55912"]
            initialDelaySeconds: 5
          livenessProbe:
            exec:
              command: ["/bin/grpc_health_probe", "-addr=localhost:55912"]
            initialDelaySeconds: 10

The important change being the readiness and liveness probe addresses. They used to just be -addr=:55912.

I don't have a valid proxy to check against to validate that part now works. Are you able to test this out?

[2ZZ]

That is looking better thanks
I'm now getting a 403

2020-02-21T23:11:11.436558Z     error   newrelic        {"err":"unexpected post response code: 403: Forbidden"}
2020-02-21T23:11:16.434143Z     error   newrelic        {"err":"unexpected post response code: 403: Forbidden"}
2020-02-21T23:11:21.431911Z     error   newrelic        {"err":"unexpected post response code: 403: Forbidden"}

Is it possible to increase logging so I know which URL is returning it? I may need to whitelist it in the proxy.

[MrAlias]

@2ZZ : you can set the log verbosity as an arg to the adapter command:

...
          args:
            - --log-level
            - debug

Probably something we should expose via a Helm value...

[MrAlias]

Probably something we should expose via a Helm value...

Nevermind, it is something we already support: https://github.com/newrelic/newrelic-istio-adapter/blob/master/helm-charts/values.yaml#L33-L35

Just need to updated the docs about it.

[2ZZ]

thanks

2020-02-21T23:19:31.464112Z     info    newrelic        {"body-length":62410,"event":"data post","url":"https://metric-api.newrelic.com/metric/v1"}
2020-02-21T23:19:31.536885Z     error   newrelic        {"err":"unexpected post response code: 403: Forbidden"}

It was due to an incorrect API key

2020-02-21T23:28:56.119145Z     info    newrelic        {"body-length":68696,"event":"data post","url":"https://metric-api.newrelic.com/metric/v1"}
2020-02-21T23:28:56.197416Z     info    newrelic        {"body":{"requestId":"679fa7b5-001d-b000-0000-01706a152785"},"event":"data post response","status":202}

all working now
thanks for the help!

[MrAlias]

@2ZZ : awesome, glad to hear!

I'm going to close this as resolved, but I created #23 to allow for easy configuration via Helm of the proxies. Please feel free to comment on that PR if you have opinions about the approach.