[BUG] agent_error: Failed to connect to localhost port 8126: Connection refused

[leonildojunior]

Bug description

I'm installing the extension to instrument a Laravel application and when I run phpinfo() I get the error agent_error:Failed to connect to localhost port 8126: Connection refused.

I'm uploading my application using docker-compose. Below is how my file is:

version: "2.4"
services:
  db:
    image: mysql:8.0
    volumes:
      - ./docker/mysql:/docker-entrypoint-initdb.d
      - ./docker/mysql/data:/var/lib/mysql
    command: ["--default-authentication-plugin=mysql_native_password"]
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: xxxxxxxx
      MYSQL_DATABASE: xxxxxxx
      MYSQL_USER: xxxxxx
      MYSQL_PASSWORD: xxxxxx
    networks:
      - internal
  php:
    build: ./docker/php-fpm
    ports:
      - "9000:9000"
    volumes:
      - .:/var/www/api:cached
      - ./docker/var/log/api:/var/www/api/var/log:cached
    links:
      - db
      - redis
      - memcached
    extra_hosts:
      - "docker.host:127.0.0.1"
    networks:
      - internal
  nginx:
    build: ./docker/nginx
    ports:
      - "4000:80"
    links:
      - php
    volumes_from:
      - php
    volumes:
      - ./docker/var/log/nginx/:/var/log/nginx:cached
    networks:
      - internal
  redis:
    image: sickp/alpine-redis:4.0.6
    ports:
      - "6380:6379"
    networks:
      - internal
    labels:
      com.datadoghq.ad.logs: '[{"source": "redis", "service": "redis"}]'
  memcached:
    image: memcached:1.6.6-alpine
    ports:
      - "11211:11211"
    networks:
      - internal
  redis_integration:
    image: "datadog/dd-trace-ci:php-redis-5.0"
  ddagent_integration:
    image: datadog/agent:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "-X", "HEAD", "http://localhost:8126"]
      interval: 10s
      timeout: 2s
      retries: 2
    environment:
      - DD_APM_ENABLED=true
      - DD_BIND_HOST=0.0.0.0
      - DD_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    ports:
      - "8126:8126"
    networks:
      - internal
  datadog:
    image: datadog/agent:latest
    links:
      - redis # Connect the Datadog Agent container to the Redis container
    environment:
      - DD_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      - DD_SITE=datadoghq.com
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_DOCKER_CONTAINER_USE_FILE=true # To read log from file instead of the docker socket
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup:/host/sys/fs/cgroup:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    networks:
      - internal

networks:
  internal:
    driver: bridge

I'm setting the environment variables in the php Dockerfile as follows:

FROM alpine:3.11

RUN apk add --update \
    php7-fpm \
    php7-apcu \
    php7-ctype \
    php7-curl \
    php7-dom \
    php7-gd \
    php7-fileinfo \
    php7-iconv \
    php7-imagick \
    php7-json \
    php7-intl \
    php7-mbstring \
    php7-opcache \
    php7-openssl \
    php7-pdo \
    php7-pdo_mysql \
    php7-pdo_sqlite \
    php7-mysqli \
    php7-xml \
    php7-zlib \
    php7-phar \
    php7-tokenizer \
    php7-session \
    php7-simplexml \
    php7-sqlite3 \
    php7-xdebug \
    php7-xmlreader \
    php7-xmlwriter \
    php7-zip \
    php7-redis \
    php7-memcached \
    make \
    curl \
    git \
    memcached \
    wget

RUN apk add bash
ENV DDTRACE_VERSION=0.60.0
RUN wget --no-check-certificate https://github.com/DataDog/dd-trace-php/releases/download/${DDTRACE_VERSION}/datadog-php-tracer_${DDTRACE_VERSION}_noarch.apk
RUN apk add --allow-untrusted ./datadog-php-tracer_${DDTRACE_VERSION}_noarch.apk

RUN touch /etc/php7/conf.d/uploads.ini \
    && echo "upload_max_filesize = 10M;" >> /etc/php7/conf.d/uploads.ini \
    && echo "max_input_vars = 5000;" >> /etc/php7/conf.d/uploads.ini \
    && echo "max_execution_time = 300;" >> /etc/php7/conf.d/uploads.ini \
    && echo "request_terminate_timeout = 300;" >> /etc/php7/conf.d/uploads.ini

RUN apk add --no-cache bash

RUN rm -rf /var/cache/apk/* && rm -rf /tmp/* && \
    curl --insecure https://getcomposer.org/composer.phar -o /usr/bin/composer && chmod +x /usr/bin/composer

ADD xdebug.ini  /etc/php7/conf.d/

ADD api.pool.conf /etc/php7/php-fpm.d/

CMD ["php-fpm7", "-F"]

WORKDIR /var/www/api
EXPOSE 9000

And the conf looks like this:

; Start a new pool named 'api'.
; the variable $pool can be used in any directive and will be replaced by the
; pool name ('api' here)
[api]

; Unix user/group of processes
; Note: The user is mandatory. If the group is not set, the default user's group
;       will be used.
user = nobody
group = nobody

; The address on which to accept FastCGI requests.
; Valid syntaxes are:
;   'ip.add.re.ss:port'    - to listen on a TCP socket to a specific address on
;                            a specific port;
;   'port'                 - to listen on a TCP socket to all addresses on a
;                            specific port;
;   '/path/to/unix/socket' - to listen on a unix socket.
; Note: This value is mandatory.
listen = 0.0.0.0:9002

; Choose how the process manager will control the number of child processes.
; Possible Values:
;   static  - a fixed number (pm.max_children) of child processes;
;   dynamic - the number of child processes are set dynamically based on the
;             following directives. With this process management, there will be
;             always at least 1 children.
;             pm.max_children      - the maximum number of children that can
;                                    be alive at the same time.
;             pm.start_servers     - the number of children created on startup.
;             pm.min_spare_servers - the minimum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is less than this
;                                    number then some children will be created.
;             pm.max_spare_servers - the maximum number of children in 'idle'
;                                    state (waiting to process). If the number
;                                    of 'idle' processes is greater than this
;                                    number then some children will be killed.
;  ondemand - no children are created at startup. Children will be forked when
;             new requests will connect. The following parameter are used:
;             pm.max_children           - the maximum number of children that
;                                         can be alive at the same time.
;             pm.process_idle_timeout   - The number of seconds after which
;                                         an idle process will be killed.
; Note: This value is mandatory.
pm = dynamic

; The number of child processes to be created when pm is set to 'static' and the
; maximum number of child processes when pm is set to 'dynamic' or 'ondemand'.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI. The below defaults are based on a server without much resources. Don't
; forget to tweak pm.* to fit your needs.
; Note: Used when pm is set to 'static', 'dynamic' or 'ondemand'
; Note: This value is mandatory.
pm.max_children = 20

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 2

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 1

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 3

;---------------------

; Make specific Docker environment variables available to PHP
env[DB_1_ENV_MYSQL_DATABASE] = $DB_1_ENV_MYSQL_DATABASE
env[DB_1_ENV_MYSQL_USER] = $DB_1_ENV_MYSQL_USER
env[DB_1_ENV_MYSQL_PASSWORD] = $DB_1_ENV_MYSQL_PASSWORD

env[DD_SERVICE] = expert-expertevent-dev
env[DD_ENV] = dev
env[DD_VERSION] = 1.0.0
env[DD_TRACE_LARAVEL_ENABLED] = true

catch_workers_output = 1

PHP version

PHP 7.3.22

Installed extensions

Step 12/15 : RUN php -m
---> Running in b4ae84d9365f
Found incompatible Xdebug version 2.9.1; ddtrace requires Xdebug 2.9.5 or greater; disabling conflicting functionality
[PHP Modules]
apcu
Core
ctype
curl
date
ddtrace
dom
fileinfo
filter
gd
hash
iconv
igbinary
imagick
intl
json
libxml
mbstring
memcached
mysqli
mysqlnd
openssl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
readline
redis
Reflection
session
SimpleXML
SPL
sqlite3
standard
tokenizer
xdebug
xml
xmlreader
xmlwriter
Zend OPcache
zip
zlib

[Zend Modules]
Xdebug
Zend OPcache
ddtrace

OS info

NAME="Alpine Linux"
VERSION_ID=3.11.11
PRETTY_NAME="Alpine Linux v3.11"

Diagnostics and configuration

Output of phpinfo() (ddtrace >= 0.47.0)

]

[SammyK]

👋 @leonildojunior The Agent will try to connect to localhost by default. Since you have the Agent running in a separate container identified by the service datadog, you will need to configure the hostname of the Agent in the php service by setting the environment variable, DDAGENT_HOSTNAME=datadog DD_AGENT_HOST=datadog.

Also, the PHP tracer will not run with Xdebug 2.9.1, so that will need to be upgraded to 2.9.5 or greater to enable the PHP tracer.

[leonildojunior]

@SammyK thank you very much for your feedback.

Even after adding the DD_AGENT_HOSTNAME you keep getting the error when connecting to the agent :(

env[DD_SERVICE] = expert-expertevent-dev
env[DD_ENV] = dev
env[DD_VERSION] = 1.0.0
env[DD_TRACE_LARAVEL_ENABLED] = true
env[DD_AGENT_HOSTNAME] = datadog

One thing I noticed, outputting getenv() in my application I retrieve the values defined in www.conf, but if I output using $_ENV[]or $_SERVER[]` I don't have the values, will it be that is something related to that?

[SammyK]

My apologies, I copied an old env var that is no longer used. In your configuration, DD_AGENT_HOSTNAME should actually be DD_AGENT_HOST.

[leonildojunior]

I made the change and continue with the same problem, the environment variables:

env[DD_SERVICE] = expert-expertevent-dev
env[DD_ENV] = dev
env[DD_VERSION] = 1.0.0
env[DD_TRACE_LARAVEL_ENABLED] = true
env[DD_AGENT_HOST] = datadog

In the laravel controller I did a test
echo getenv('DD_AGENT_HOST');
The output was: datadog

[labbati]

can you ping datadog from the php container? I think you have to add either datadog or ddagent_integration (the one you want to use), to the links section of the php service.

[leonildojunior]

Inside the PHP Docker container I can ping datadog as shown in the print, but without the port and the IP I don't know if it's calling localhost which is the default. I added datadog to the PHP links as shown in the code below but it still gives an error in phpinfo():

php:
     build: ./docker/php-fpm
     ports:
       - "9000:9000"
     volumes:
       - .:/var/www/api:cached
       - ./docker/var/log/api:/var/www/api/var/log:cached
     links:
       - db
       - nets
       - memcached
       - datadog
     extra_hosts:
       - "docker.host:127.0.0.1"
     networks:
       - internal

[leonildojunior]

Analyzing the datadog/agent container log I found a line with the error below, maybe it will help, but I don't know how to analyze it, I don't understand the error, but it appears at various times between the various INFO

2021-06-28 22:22:58 UTC | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:108 in func4) | TLS Handshake failure: net/http: TLS handshake timeout

2021-06-28 22:22:58 UTC | CORE | ERROR | (pkg/forwarder/worker.go:183 in process) | Error while processing transaction: error while sending transaction, rescheduling it: Post "https://7-29-0-app.agent.datadoghq.com/intake/?api_key=************ ***************a5cb9": net/http: TLS handshake timeout

Besides the error I have some WARN that follows below, other than that they are just INFO I don't know to what extent it would be relevant for the analysis.

2021-06-28 22:22:31 UTC | SYS PROBE | WARN | (pkg/util/log/log.go:541 in func1) | Error loading config: open /etc/datadog-agent/system-probe.yaml: no such file or directory

2021-06-28 22:22:31 UTC | PROCESS | WARN | (pkg/util/log/log.go:541 in func1) | failed to get configuration value for key "process_agent_config.host_ips": unable to cast <nil> of type <nil> to []string

2021-06-28 22:22:38 UTC | CORE | WARN | (pkg/logs/client/http/destination.go:256 in CheckConnectivity) | HTTP connectivity failure: Post "https://agent-http-intake.logs.datadoghq.com/v1/input": context deadline exceeded (Client.Timeout exceeded while waiting headers)

2021-06-28 22:22:38 UTC | CORE | WARN | (pkg/logs/config/config.go:119 in BuildEndpointsWithConfig) | You are currently sending Logs to Datadog through TCP (either because logs_config.use_tcp or logs_config.socks5_proxy_address is set or HTTP connectivity test has failed) To benefit from increased reliability and better network performances, we strongly encourage switching over to compressed HTTPS which is now the default protocol.

2021-06-28 22:22:40 UTC | CORE | WARN | (pkg/logs/scheduler/scheduler.go:78 in Schedule) | Invalid service: entity is malformed :

021-06-28 22:22:48 UTC | CORE | WARN | (pkg/util/gce/gce_tags.go:48 in getCachedTags) | unable to get tags from gce and cache is empty: Get "http://169.254.169.254/computeMetadata/v1/?recursive=true": context deadline exceeded (Client.Timeout exceeded while waiting headers)

2021-06-28 22:22:50 UTC | CORE | WARN | (pkg/config/viper.go:189 in GetStringSlice) | failed to get configuration value for key "process_agent_config.host_ips": unable to cast <nil> of type <nil> to []string

[gnumoksha]

Same issue here. It worked fine until I had to download the agent from docker-hub again. Now I'm using the image datadog/agent:7.28.0 and dd-trace-php 0.60 (but I don't know if this is related to the issue) and it is working as expected.

[aleferreiranogueira]

+1 on @gnumoksha , same dd-trace version

[juztin]

Ughh, same boat.
New to DD, and have been pounding my head on documentation that is very lacking.

I've got the otelcol-contrib running, and it appears to be working (have no idea how to test, other than the logs saying it's okay)
Using the hard to find instructions on DD's trace provider, and that appears okay as well, but when the application starts up, it complains about not being able to connect to localhost:8216. The otelcol-contrib logs show ports :4317 (GRPC) and :4318 (HTTP) being open.

Was able to wade through a bunch of docs, and set this ENV variable: DD_TRACE_AGENT_PORT=4318

Everything spins up and runs, I'm not seeing any errors in systemd logs.

The otelcol-contrib does have these in the logs:

May 12 20:54:03 sample_service otelcol-contrib[13151]: 2023-05-12T20:54:03.102Z        warn        ec2/ec2.go:76        Failed to get EC2 instance id document        {"kind": "exporter", "data_type": "metrics", "name": "datadog", "error": "SerializationError: failed to decode EC2 instance identity document\ncaused by: EOF"}

while the service containing the OTEL tracing is spitting out these during spans:

May 12 21:32:31 sample_service sample_service[13591]: 2023/05/12 21:32:31 Datadog Tracer v1.50.1 INFO: DATADOG TRACER CONFIGURATION {"date":"2023-05-12T21:32:31Z","os_name":"Debian GNU/Linux","os_version":"11 (bullseye)","version":"v1.50.1","lang":"Go","lang_version":"go1.20.3","env":"prod","service":"sample_service","agent_url":"http://localhost:4318/v0.4/traces","agent_error":"","debug":false,"analytics_enabled":false,"sample_rate":"NaN","sample_rate_limit":"disabled","sampling_rules":null,"sampling_rules_error":"","service_mappings":null,"tags":{"runtime-id":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"},"runtime_metrics_enabled":false,"health_metrics_enabled":false,"profiler_code_hotspots_enabled":true,"profiler_endpoints_enabled":true,"dd_version":"0.0.0","architecture":"amd64","global_service":"sample_service","lambda_mode":"false","appsec":false,"agent_features":{"DropP0s":false,"Stats":false,"StatsdPort":0}}

So it appears as though it should be working, but no data shows up within the DD panel, so I'm unsure if it's swallowing errors, or just plain not working. If I swap out the DD provider and simply use a one that uses stdout, all traces/spans show up perfectly. Anyone running into anything similar/have any pointers?

[dhanakane-vc]

Using the hard to find instructions on DD's trace provider, and that appears okay as well, but when the application starts > up, it complains about not being able to connect to localhost:8216.

Maybe a typo in your comment, but is the port number you're using correct? I think it's supposed to be 8126. FWIW, I've started seeing this issue too, and I cannot get a successful message using the healthcheck endpoint on localhost to that port when running it locally in a container (it returns a 404 error).