datadog agent for docker linux stops unexpectedly

[ondiekisteven]

Hi, i am new to docker and i followed this youtube video to run the docker agent container using this command docker run -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e DD_API_KEY=XXXX -e DD_SITE="datadoghq.com" gcr.io/datadoghq/agent:7. However the container stops after few seconds everytime. So i tried running it without the -d option and this is the output:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-check-apikey.sh: executing... 
[cont-init.d] 01-check-apikey.sh: exited 0.
[cont-init.d] 50-ci.sh: executing... 
[cont-init.d] 50-ci.sh: exited 0.
[cont-init.d] 50-ecs.sh: executing... 
[cont-init.d] 50-ecs.sh: exited 0.
[cont-init.d] 50-eks.sh: executing... 
[cont-init.d] 50-eks.sh: exited 0.
[cont-init.d] 50-kubernetes.sh: executing... 
[cont-init.d] 50-kubernetes.sh: exited 0.
[cont-init.d] 50-mesos.sh: executing... 
[cont-init.d] 50-mesos.sh: exited 0.
[cont-init.d] 51-docker.sh: executing... 
[cont-init.d] 51-docker.sh: exited 0.
[cont-init.d] 59-defaults.sh: executing... 
[cont-init.d] 59-defaults.sh: exited 0.
[cont-init.d] 60-network-check.sh: executing... 
[cont-init.d] 60-network-check.sh: exited 0.
[cont-init.d] 60-sysprobe-check.sh: executing... 
grep: /etc/datadog-agent/system-probe.yaml: No such file or directory
grep: /etc/datadog-agent/system-probe.yaml: No such file or directory
[cont-init.d] 60-sysprobe-check.sh: exited 0.
[cont-init.d] 89-copy-customfiles.sh: executing... 
[cont-init.d] 89-copy-customfiles.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
starting process-agent
starting trace-agent
starting agent
starting system-probe
starting security-agent
[services.d] done.
2022-01-04 02:53:22 UTC | CORE | INFO | (pkg/util/log/log.go:610 in func1) | runtime: final GOMAXPROCS value is: 1
2022-01-04 02:53:22 UTC | CORE | INFO | (pkg/util/log/log.go:610 in func1) | Features detected from environment: docker
2022-01-04 02:53:22 UTC | CORE | INFO | (cmd/agent/app/run.go:248 in StartAgent) | Starting Datadog Agent v7.32.4
2022-01-04 02:53:22 UTC | PROCESS | INFO | (pkg/util/log/log.go:610 in func1) | Features detected from environment: docker
2022-01-04 02:53:22 UTC | PROCESS | INFO | (pkg/util/log/log.go:610 in func1) | Collector docker successfully detected
2022-01-04 02:53:22 UTC | PROCESS | INFO | (pkg/util/log/log.go:610 in func1) | Using collector docker
2022-01-04 02:53:22 UTC | PROCESS | WARN | (pkg/util/log/log.go:630 in func1) | failed to get configuration value for key "process_agent_config.host_ips": unable to cast <nil> of type <nil> to []string
2022-01-04 02:53:22 UTC | PROCESS | INFO | (pkg/util/log/log.go:610 in func1) | could not deduce host IP from source config: no hostIPs were configured
2022-01-04 02:53:22 UTC | PROCESS | INFO | (pkg/util/log/log.go:610 in func1) | could not deduce host IP from source ec2 metadata endpoint: unable to fetch EC2 API, Get "http://169.254.169.254/latest/meta-data/local-ipv4": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-01-04 02:53:22 UTC | PROCESS | INFO | (pkg/util/log/log.go:610 in func1) | overriding API key from env DD_API_KEY value
2022-01-04 02:53:22 UTC | PROCESS | INFO | (cmd/system-probe/config/config.go:119 in Merge) | no config exists at /etc/datadog-agent/system-probe.yaml, ignoring...
2022-01-04 02:53:24 UTC | CORE | ERROR | (cmd/agent/app/run.go:315 in StartAgent) | Error while getting hostname, exiting: unable to reliably determine the host name. You can define one in the agent config file or in your hosts file
2022-01-04 02:53:24 UTC | CORE | INFO | (pkg/logs/logs.go:174 in Stop) | Stopping logs-agent
2022-01-04 02:53:24 UTC | CORE | INFO | (pkg/logs/logs.go:186 in Stop) | logs-agent stopped
2022-01-04 02:53:24 UTC | CORE | INFO | (cmd/agent/app/run.go:519 in StopAgent) | See ya!
Error: Error while getting hostname, exiting: unable to reliably determine the host name. You can define one in the agent config file or in your hosts file
AGENT EXITED WITH CODE 255, SIGNAL 0, KILLING CONTAINER
AGENT EXITED WITH CODE 256, SIGNAL 15, KILLING CONTAINER
process-agent exited with code 256, signal 15, restarting in 2 seconds
security-agent exited with code 256, signal 15, restarting in 2 seconds
system-probe exited with code 256, signal 15, restarting in 2 seconds
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/util/log/log.go:610 in func1) | runtime: final GOMAXPROCS value is: 1
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/util/log/log.go:610 in func1) | Features detected from environment: docker
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/util/log/log.go:610 in func1) | Loaded configuration: /etc/datadog-agent/datadog.yaml
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/util/log/log.go:610 in func1) | received signal 15 (terminated)
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:257 in startCandidates) | workloadmeta collector "kubelet" could not start. error: the Agent is not running in Kubernetes
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/workloadmeta/store.go:85 in Start) | workloadmeta store initialized successfully
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/tagger/local/tagger.go:180 in tryCollectors) | collector cloudfoundry skipped as feature not activated
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/tagger/local/tagger.go:180 in tryCollectors) | collector kube-metadata-collector skipped as feature not activated
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/tagger/local/tagger.go:191 in tryCollectors) | workloadmeta tag collector successfully started
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/tagger/local/tagger.go:191 in tryCollectors) | docker tag collector successfully started
2022-01-04 02:53:26 UTC | TRACE | INFO | (pkg/tagger/local/tagger.go:180 in tryCollectors) | collector ecs_fargate skipped as feature not activated
s6-svwait: fatal: supervisor died
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

Describe what you expected:
i expected the container to run and send data to the dashboard but it just stopped

Additional environment details (Operating System, Cloud provider, etc):
I am running ubuntu on Linode
$ lsb_release -a

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

I am not sure what could be the issue. Any help will be appreciated. Thanks

[sgnn7]

Hey @ondiekisteven,
I have had no problems running the 7.33.x agent on 20.04.3 LTS using this snippet:

docker run --rm -it \
      --name dd-agent \
      -e DD_API_KEY="<key>" \
      -e DD_SITE="datadoghq.com" \
      -e DD_HOSTNAME="<hostname>" \
      -e DD_LOG_LEVEL="info" \
      -v /proc/:/host/proc/:ro \
      -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
      -v /var/run/docker.sock:/var/run/docker.sock:ro \
      -v /opt/datadog-agent/run:/opt/datadog-agent/run:rw \
      gcr.io/datadoghq/agent:7

Can you give this a shot (you can remove the --rm -it and use -d for your use case) and let us know if that works?

[sgnn7]

@ondiekisteven Since we haven't heard from you in a while on this, I will assume that the issue is resolved and will close it. Feel free to reopen it if needed.

[ronindesign]

I also just ran into this issue using default agent install and config on Fedora 37. Linux 6.0.8-300.fc37.x86_64