datadoghq.eu supported?

[onpaws]

Output of the info page (if this is a bug)
Not sure if this is a bug - I think this is more of a case where the proper way to specify something isn't clear (at least not clear to me). I think I need to specify the EU 'flavor' of Datadog to the Operator, and not sure if the needed config surface area already exists, or if it does exist, where it is.

Describe what happened:
As I understand it Datadog users can create their account in the US or the EU.
In my case I created my account on the EU side, i.e. I pull up the dashboard etc at https://app.datadoghq.eu/

Following the instructions here
https://docs.datadoghq.com/agent/kubernetes/?tab=operator
I generated an API key and App key, and installed the datadog-operator using Helm 3.

Describe what you expected:
I was expecting the usual data flow to work...i.e. cluster data/etc to flow into https://app.datadoghq.eu/, but when I check status of the deploy, the dd resource never reaches 'ACTIVE' status after 30 minutes. Ostensibly this is because the agents are pointing to datadoghq.com not .eu. Not 100% sure.

$ kubectl  -n datadog get dd
NAME                                     ACTIVE   AGENT   CLUSTER-AGENT   CLUSTER-CHECKS-RUNNER   AGE
datadog-agent-with-operator-1596710136                                                            30m

If I understand right this may be a related fix? #104
I was using this release
https://github.com/DataDog/datadog-operator/releases/tag/v0.2.1

Steps to reproduce the issue:
The above covers it, but in a sentence, try to use https://github.com/DataDog/datadog-operator/releases/tag/v0.2.1 on a .EU hosted Datadog account.

Additional environment details (Operating System, Cloud provider, etc):
Azure AKS, macOS 10.15.6, Helm 3, datadog-operator v0.2.1

[cohenyair]

Hi @onpaws, Can you try to use the param site: "datadoghq.eu" in DatadogAgent CRD as documented in this page: https://github.com/DataDog/datadog-operator/blob/master/docs/configuration.md? Please let us know if it doesn't solve your issue

[onpaws]

Thanks @cohenyair !
I will give it a try and post back. I'm attending KubeCon this week so will probably be a few days :)

[onpaws]

Ok, I think i just repro'd. Was trying to deploy the Datadog Operator using Helm using the directions here

datadog-operator pod logs:

2020-08-21T00:16:44.816Z	ERROR	DatadogMetricForwarders	cannot get Datadog metrics forwarder to send deployment metrics, will retry later...	{"CustomResource.Namespace": "datadog", "CustomResource.Name": "datadog-datadog-agent-with-operator", "error": "invalid datadog credentials"}
github.com/go-logr/zapr.(*zapLogger).Error
	/src/vendor/github.com/go-logr/zapr/zapr.go:128
github.com/DataDog/datadog-operator/pkg/controller/utils/datadog.(*metricsForwarder).connectToDatadogAPI
	/src/pkg/controller/utils/datadog/metrics_forwarder.go:210
k8s.io/apimachinery/pkg/util/wait.WaitFor
	/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:434
k8s.io/apimachinery/pkg/util/wait.PollUntil
	/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:385
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil
	/src/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:404
github.com/DataDog/datadog-operator/pkg/controller/utils/datadog.(*metricsForwarder).start
	/src/pkg/controller/utils/datadog/metrics_forwarder.go:129
2020-08-21T00:16:44.994Z	INFO	DatadogAgent	Reconciling DatadogAgent	{"Request.Namespace": "datadog", "Request.Name": "datadog-datadog-agent-with-operator"}
2020-08-21T00:16:45.016Z	INFO	DatadogAgent	Reconciling DatadogAgent	{"Request.Namespace": "datadog", "Request.Name": "datadog-datadog-agent-with-operator"}
2020-08-21T00:16:45.034Z	INFO	DatadogAgent	Reconciling DatadogAgent	{"Request.Namespace": "datadog", "Request.Name": "datadog-datadog-agent-with-operator"}

datadog-agent.yaml:

credentials:
  apiKey: <snip>
  appKey: <snip>
agent:
  image:
    name: "datadog/agent:latest"
site: "datadoghq.eu"

helm install -n datadog --set-file agent_spec=datadog-agent.yaml datadog datadog-agent-with-operator.tar.gz --dry-run

# Source: datadog-agent-with-operator/templates/datadog-agent.yaml
apiVersion: datadoghq.com/v1alpha1
kind: DatadogAgent
metadata:
  name: datadog-datadog-agent-with-operator
  labels:
    helm.sh/chart: datadog-agent-with-operator-0.2.1
    app.kubernetes.io/name: datadog-agent-with-operator
    app.kubernetes.io/instance: datadog
    app.kubernetes.io/version: "0.2.1"
    app.kubernetes.io/managed-by: Helm
spec:
  credentials:
    apiKey: snip
    appKey: snip
  agent:
    image:
      name: "datadog/agent:latest"
  site: "datadoghq.eu"

NOTES:
Datadog agents are spinning up on each node in your cluster. After a few
minutes, you should see your agents starting in your event stream:
    https://app.datadoghq.com/event/stream

[onpaws]

No data flows into my Datadog so I think I'm still at the same spot.
Feels like I'm probably missing something...?
Please let me know if you need any more logs/etc.

Passing the same creds into the non Operator Helm chart works, so we should be able to be confident the apiKey is valid.

[mantoine96]

@onpaws FYI I've encountered the same issues as you did, and created several PRs that should in theory also address your problem:
#104
#92
#90

These changes should be released as part of 0.3.0. I'm currently running a custom build where I only set site: datadoghq.eu and the monitoring data is sent to the EU endpoint as expected.

To have 0.2.1 work properly requires specifying environment variables to each pod of a datadog spec, such as DATADOG_HOST or DD_DD_URL.

[onpaws]

Thanks for posting back @thehunt33r ! Appreciate the follow up.
Looks like 0.3.0 shipped, right?
Would you say it's ready for me to try, see if it solves my issue?

[mantoine96]

You're right it shipped last week! I would definitely recommend you give it a try!

[onpaws]

Thanks.
I just tried to uninstall my old Operator, which worked in the end, but I had to manually patch out the 'Finalizer' as seen here:

k -n datadog describe datadogagent datadog-datadog-agent-with-operator
Name:         datadog-datadog-agent-with-operator
Namespace:    datadog
Labels:       app.kubernetes.io/instance=datadog
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=datadog-agent-with-operator
              app.kubernetes.io/version=0.2.1
              helm.sh/chart=datadog-agent-with-operator-0.2.1
Annotations:  meta.helm.sh/release-name: datadog
              meta.helm.sh/release-namespace: datadog
API Version:  datadoghq.com/v1alpha1
Kind:         DatadogAgent
Metadata:
...
  Finalizers:
    finalizer.agent.datadoghq.com
  Generation:  4
...

Having tackled similar issues in the past trying to delete namespaces I'm glad I managed in the end by patching out the finalizer with kubectl edit.
I'm curious, if you don't mind what is the intended purpose of the finalizer here?

[onpaws]

I'm going to close this issue because my team prefers to go with the 'known path' Helm chart this time around. Thanks for your efforts on the Operator Datadog team, future me looks forward to trying again next time

[txomon]

As a tip for anyone that also decided to go with the 'known path', you are probably going to hit the same problem, but in the cluster agent. Relevant link: DataDog/datadog-agent#6066