auth_token unreadable and agent won't start after enabling docker_daemon integration

[eedwards-sk]

Output of the info page (if this is a bug)
from agent.log (info will not run due to service not starting)

2018-04-05 16:28:07 UTC | INFO | (start.go:156 in StartAgent) | Starting Datadog Agent v6.1.1
2018-04-05 16:28:07 UTC | INFO | (start.go:167 in StartAgent) | pid '9456' written to pid file '/opt/datadog-agent/run/agent.pid'
2018-04-05 16:28:07 UTC | INFO | (start.go:174 in StartAgent) | Hostname is: rancher-admin-eng-i-0c9526fd8bffdf0f9
2018-04-05 16:28:07 UTC | ERROR | (start.go:185 in StartAgent) | Error while starting api server, exiting: unable to access authentication token: open /etc/datadog-agent/auth_token: permission denied
2018-04-05 16:28:07 UTC | INFO | (start.go:322 in StopAgent) | See ya!

Describe what happened:
datadog agent created the auth_token file with a user of root and group of root, and is unable to read it as user dd-agent

Describe what you expected:
datadog agent created the auth_token file with a user of dd-agent and group of dd-agent, and is able to read it as user dd-agent

Steps to reproduce the issue:
this worked before enabling the docker integration
once I added this to my setup procedure:

  # add dd-agent user to docker group
  usermod -a -G docker dd-agent
  # download datadog integration files and parse any variables
  aws s3 cp s3://${S3_BUCKET}/config/${project_name}/all/etc/datadog-agent/conf.d/docker.d/conf.yaml /etc/datadog-agent/conf.d/docker.d/conf.yaml
  chown dd-agent:dd-agent /etc/datadog-agent/conf.d/docker.d/conf.yaml

and added this docker config:

init_config:

instances:
  - ## Daemon and system configuration
    url: "unix://var/run/docker.sock"

then it no longer worked

deleting the /etc/datadog-agent/auth_token file and restarting the datadog agent caused it to be created correctly with an owner of dd-agent:dd-agent

Additional environment details (Operating System, Cloud provider, etc):
ec2 ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170619.1 (ami-d15a75c7)
docker 1.12 / rancher

[eedwards-sk]

I tried the process again and this time it worked.

I'll close this, but appending some info for posterity:

I believe the system was still in the process of installing / configuring when I SSHd in and ran sudo datadog-agent status

It appears that the datadog-agent status command when ran prior to the installation finishing and service being started, causes the auth_token to be created as the user running the command (root in this case) rather than the user that would run the service (dd-agent)

[ragnarkurmwunder]

Thanks, @eedwards-sk pointing to the right direction!

Just adding my experience, and confirmation, for anyone who stumbles upon the auth_token problem in the context of Datadog 7.
I had to make sure that /etc/datadog-agent and /etc/datadog-agent/auth_token both have ownership dd-agent.
The problem happens when the agent is not running, but running datadog-agent status manually as root from command line, which creates the auth_token owned by root.

Here is my start script:

#!/bin/bash

set -xeuo pipefail
export PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin"

# Need to ensure proper ownership
# or things start to fail later
readonly token=/etc/datadog-agent/auth_token

# .ebextensions/datadog/hooks/99start_datadog.sh
if [[ ! -e /etc/init/datadog-agent.conf ]]; then
  echo "Agent not installed"
  exit 0
fi

# The Datadog conf dir is also used for keeping state: the auth token.
# Datadog agent must be able to write there as dd-agent.
# Therefore the dir needs to have dd-agent ownership.
chown -v dd-agent $(dirname "$token")

readonly stat=$(initctl status datadog-agent)
if [[ "$stat" == *"datadog-agent start/running"* ]]; then
  echo "Agent already running"
  exit 0
fi

initctl start datadog-agent

# Allow some time to write down the token.
sleep 1

# Make sure the token is owned by dd-agent, or the agent fails.
# Usually the token will have that ownership, but with on exception:
# If the Datadog is not running and the command 'datadog-agent'
# is manually run from the command line, then it creates the token
# with root ownership, causing failures down the line.
if [[ -f "$token" ]]; then
  chown -v dd-agent "$token"
fi

[albertvaka]

Posting here for completion. This was fixed in #5064