ddev does not respect connections to external networks anymore

[galoppi]

Is there an existing issue for this?

• I have searched the existing issues

Output of ddev debug test

Expand `ddev debug test` diagnostic information

Running bash [-c /tmp/test_ddev.sh]
======= Existing project config =========
These config files were loaded for project 726-feature-84358-testomize-php8-2: [/home/gitlab-runner/builds/mysysmex-mobifi/feature-84358-testomize-php8-2/.ddev/config.yaml /home/gitlab-runner/builds/mysysmex-mobifi/feature-84358-testomize-php8-2/.ddev/config.override.yaml]
name: 726-feature-84358-testomize-php8-2
type: typo3
docroot: public
php_version: 8.2
webserver_type: apache-fpm
webimage: ddev/ddev-webserver:v1.22.6
router_http_port: 8080
router_https_port: 4430
additional_fqdns: [www.mysysmex.vmdev testing.mysysmex.vmdev]
database: {mariadb 10.2}
hooks: map[pre-start:[map[exec-host:docker network inspect mysysmex >/dev/null 2>&1 || docker network create --driver bridge mysysmex]]]
project_tld: ddev.site
use_dns_when_possible: true
timezone: Europe/Berlin
composer_version: 2
nodejs_version: 16
default_container_timeout: 120
======= Creating dummy project named  tryddevproject-18316 in ../tryddevproject-18316 =========
OS Information: Linux ddev-runner-mysysmexmobifi 6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
User information: uid=1001(gitlab-runner) gid=1004(gitlab-runner) groups=1004(gitlab-runner),1002(docker)
DDEV version:  ITEM             VALUE
 DDEV version     v1.22.6
 architecture     amd64
 db               ddev/ddev-dbserver-mariadb-10.4:v1.22.6
 ddev-ssh-agent   ddev/ddev-ssh-agent:v1.22.6
 docker           24.0.7
 docker-compose   v2.23.3
 docker-platform  linux-docker
 mutagen          0.17.2
 os               linux
 router           ddev/ddev-traefik-router:v1.22.6
 web              ddev/ddev-webserver:v1.22.6
PROXY settings: HTTP_PROXY='' HTTPS_PROXY='' http_proxy='' NO_PROXY=''
======= DDEV global info =========
Global configuration:
instrumentation-opt-in=false
omit-containers=[]
performance-mode=none
router-bind-all-interfaces=false
internet-detection-timeout-ms=3000
disable-http2=false
use-letsencrypt=false
letsencrypt-email=
table-style=default
simple-formatting=false
use-hardened-images=false
fail-on-hook-fail=false
required-docker-compose-version=v2.23.3
use-docker-compose-from-path=false
project-tld=
xdebug-ide-location=
no-bind-mounts=false
router=traefik
wsl2-no-windows-hosts-mgt=false
router-http-port=80
router-https-port=443
mailpit-http-port=8025
mailpit-https-port=8026
traefik-monitor-port=10999

======= DOCKER info =========
docker location: -rwxr-xr-x 1 root root 35939040 Oct 26 11:07 /usr/bin/docker
docker version:
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:07:41 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
DOCKER_DEFAULT_PLATFORM=notset
======= Mutagen Info =========
======= Docker Info =========
Docker platform: linux-docker
Using Docker context: default (unix:///var/run/docker.sock)
docker-compose: v2.23.3
Using DOCKER_HOST=unix:///var/run/docker.sock
Docker version: 24.0.7
Able to run simple container that mounts a volume.
Able to use internet inside container.
Docker disk space:
Filesystem                Size      Used Available Use% Mounted on
overlay                  18.7G     10.6G      7.3G  59% /

The ddev-ssh-agent container has been removed. When you start it again you will have to use 'ddev auth ssh' to provide key authentication again.
Network ddev_default removed
Existing docker containers:
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
Network ddev_default created
Starting tryddevproject-18316...
Network ddev-tryddevproject-18316_default created
 Container ddev-ssh-agent  Created
 Container ddev-ssh-agent  Started
ssh-agent container is running: If you want to add authentication to the ssh-agent container, run 'ddev auth ssh' to enable your keys.
Building project images...
Project images built in 0s.
 Container ddev-tryddevproject-18316-web  Created
 Container ddev-tryddevproject-18316-db  Created
 Container ddev-tryddevproject-18316-web  Started
 Container ddev-tryddevproject-18316-db  Started
Waiting for web/db containers to become ready: [web db]
Starting ddev-router if necessary...
 Container ddev-router  Created
 Container ddev-router  Started
Waiting for additional project containers to become ready...
All project containers are now ready.
Successfully started tryddevproject-18316
Project can be reached at https://tryddevproject-18316.ddev.site https://127.0.0.1:32776
======== Curl of site from inside container:
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 17 Jan 2024 17:22:19 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

======== curl -I of http://tryddevproject-18316.ddev.site from outside:
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Wed, 17 Jan 2024 17:22:19 GMT
Server: nginx
Vary: Accept-Encoding

======== full curl of http://tryddevproject-18316.ddev.site from outside:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-18316'
======== Project ownership on host:
drwxr-xr-x 4 gitlab-runner gitlab-runner 4096 Jan 17 18:21 ../tryddevproject-18316
======== Project ownership in container:
drwxr-xr-x 4 gitlab-runner gitlab-runner 4096 Jan 17 17:21 /var/www/html
======== In-container filesystem:
Filesystem     Type 1K-blocks     Used Available Use% Mounted on
/dev/sda1      ext4  19610420 11282144   7484256  61% /var/www/html
======== curl again of tryddevproject-18316 from host:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-18316'
Thanks for running the diagnostic. It was successful.
Please provide the output of this script in a new gist at gist.github.com
Running ddev launch in 5 seconds
/usr/bin/xdg-open: 882: www-browser: not found
/usr/bin/xdg-open: 882: links2: not found
/usr/bin/xdg-open: 882: elinks: not found
/usr/bin/xdg-open: 882: links: not found
/usr/bin/xdg-open: 882: lynx: not found
/usr/bin/xdg-open: 882: w3m: not found
xdg-open: no method available for opening 'https://tryddevproject-18316.ddev.site'
Failed to run launch ; error=exit status 3

Please run cleanup after debugging with 'ddev debug testcleanup'
Failed running test_ddev.sh: exit status 1
. You can run it manually with `curl -sL -O https://raw.githubusercontent.com/ddev/ddev/master/cmd/ddev/cmd/scripts/test_ddev.sh && bash test_ddev.sh`

Expected Behavior

We added an extended docker-compose network configuration in .ddev/docker-compose.testomize-frontproxy.yaml to add ddev container to an external traefik frontproxy which uses the docker network frontproxy:

version: '3.6'

services:

  web:
    networks:
      - default
      - frontproxy
    labels:
      - "traefik.enable=true"

      - "traefik.http.routers.router-726-feature-84358-testomize-php8-2-1.rule=Host(`feature-84358-testomize-php8-2.review.example.com`)"
      - "traefik.http.routers.router-726-feature-84358-testomize-php8-2-1.entrypoints=websecure"
      - "traefik.http.routers.router-726-feature-84358-testomize-php8-2-1.tls.certresolver=basic"
      - "traefik.http.routers.router-726-feature-84358-testomize-php8-2-1.tls.domains[0].main=review.example.com"
      - "traefik.http.routers.router-726-feature-84358-testomize-php8-2-1.tls.domains[0].sans=*.review.example.com"
      - "traefik.http.routers.router-726-feature-84358-testomize-php8-2-1.middlewares=auth-726-feature-84358-testomize-php8-2-1"
      - "traefik.http.routers.router-726-feature-84358-testomize-php8-2-1.service=service-726-feature-84358-testomize-php8-2-1"
      - "traefik.http.services.service-726-feature-84358-testomize-php8-2-1.loadbalancer.server.port=80"
networks:
    frontproxy:
        name: frontproxy
        external: true

After starting the ddev project all labels are added to the docker container but it is not added to the external network frontproxy as in earlier versions of ddev.

docker inspect ddev-726-feature-84358-testomize-php8-2-web shows only the default ddev networks ddev-726-feature-84358-testomize-php8-2_default and ddev_default but NOT frontproxy.
This results in a gateway timeout error of the external front proxy because it is not able to connect to the ddev webserver container.

I would expect that the ddev container is added to the external docker network frontproxy.

Actual Behavior

The ddev containers are not connected to the external networks anymore (it was working in older versions).

Steps To Reproduce

Add a docker network configuration with connection to an external docker network in .ddev/docker-compose-something.yaml, start the setup with ddev start and inspect the ddev web container with docker inspect CONTAINERNAME. Watch out for the networks section and if the external network is listed there.

Anything else?

No response

[stasadev]

Deleted my previous reply (I thought this was caused by something else).

Found the reason for this in https://github.com/ddev/ddev/pull/3620/files#r815353519

• Change docker-compose yaml to insert networks stanza, fixes #3587 #3620

Can you still override this with your own networks stanza?

No, this replaces the networks stanza unconditionally. Agreed it's a weakness; I tried not to and failed. I haven't ever heard of anybody needing to alter networks though.

@galoppi, you seem to be the first to notice this change.

The behavior for networks was changed in v1.19.0.

[rfay]

Can you just...

• run your external traefik in the ddev_default network, so it doesn't have to connect to another one?
• Add configuration to DDEV's traefik to do whatever it is you want to do?

It's unlikely that we would fiddle with DDEV's setup to solve your unusual situation, which probably has lots of other possible solutions. We can talk about other solutions if you like. Yours sounds complex.

[galoppi]

Thanks a lot for your help and support!
Indeed we have a complex setup: We create an instant ddev setup during our CI pipeline for each merge request on a cloud server with an "offical" dynamic domain name and a letsencrypt certificate. The hostnames are dynamic and created during the CI process. We can run more than one ddev setup on the same machine in parallel, also for the same origin project without interfearing each other. Therefore we add dynamic configuration, i.e. .ddev/docker-compose.testomize.yaml containing all the information for the dynamic hostnames as labels for traefik.

I'll give it a try using the ddev_default network.

Do you have an example for using the build in traefik in combination with letsencrypt?

[rfay]

We haven't explored Let's Encrypt with traefik yet, see https://ddev.readthedocs.io/en/latest/users/topics/hosting/

But the docs link to Traefik's technique, https://doc.traefik.io/traefik/https/acme/ - your exploration would be welcome!

My own hosting situation still uses the nginx-proxy router, which still works fine with Let's Encrypt.

[rfay]

It sounds like you already must be an expert with LE and Traefik so would love to have you explore it. https://ddev.readthedocs.io/en/latest/users/extend/traefik-router/#traefik-configuration

[rfay]

No response, closing, happy to help you explore traefik, and especially happy to see you explore LetEncrypt there.