[BUG]? !reset & !override tags are not retained in merged configs

[liamjones]

Description

I don't know if this is technically a bug but it does result in some usage friction for us so I'm querying the behaviour.

!reset and !override tags get 'lost' when merging configs. This is fine when generating a complete config, but it doesn't work as expected if you're looking to generate an compose.override.yml overlay that will be automatically applied on top of a compose.yml.

I understand why but I'm wondering if it should work differently (or if there should be a command line option to opt-in to the retention of these tags).

Steps To Reproduce

With the following files:

compose.yml:

services:
  a:
    image: nginx
    ports:
      - "12345:80"
  b:
    image: nginx
    ports:
      - "12346:80"
  c:
    image: nginx
    ports:
      - "12347:80"

unbind-a.yml:

services:
  a:
    ports: !reset []

add-d.yml:

services:
  d:
    image: nginx
    ports:
      - "12348:80"
    links:
      - b

If I do docker compose -f unbind-a.yml -f add-d.yml config --no-consistency > compose.override.yml the outputted compose-override will be:

name: docker-compose-config-merge-test
services:
  a:
    networks:
      default: null
  d:
    depends_on:
      b:
        condition: service_started
        restart: true
        required: true
    image: nginx
    links:
      - b
    networks:
      default: null
    ports:
      - mode: ingress
        target: 80
        published: "12348"
        protocol: tcp
networks:
  default:
    name: docker-compose-config-merge-test_default

docker compose up, etc will work as expected for adding service d to the environmentbut service a will still bind ports because the base compose.yml contains a port definition and the !reset isn't retained in the override's generated config.

Compose Version

Docker Compose version v2.29.1-desktop.1

Docker Environment

Client:
 Version:    27.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1-desktop.1
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1-desktop.1
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.11.0
    Path:     /Users/liam.jones/.docker/cli-plugins/docker-scout

Server:
 Containers: 29
  Running: 15
  Paused: 0
  Stopped: 14
 Images: 363
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.0-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 11.68GiB
 Name: docker-desktop
 ID: ebcb7a0f-bf2e-4d02-9295-c44e58f8dd69
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/liam.jones/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Anything else?

No response

[jhrotko]

Hello, @liamjones
I am wondering why would you want to create an override overlay when you can achieve the final and correct configuration by running docker compose -f compose.yml -f unbind-a.yml -f add-d.yml config --no-consistency

[liamjones]

Hi @jhrotko,

Mostly to reduce friction; not having to repeat a myriad of -fs on every command & not having to keep regenerating the config when altering the main compose.yml in a way that isn't affected by the others in the override layer.

[liamjones]

I appreciate this may be a "you're using this in a way other than intended" issue! 😄

[jhrotko]

For your use case it seems you need to have a "reset/override" layer when applying to the overlay (as you metioned) layer. Would that work?
I assume your real case is more complicated than the sample you provided (possibly with more layers), and in this case you would need to apply all the resets/ovverides - in your case unbind-a.yml - at the end of the merge sequence. Implementing a custom tag for this case sounds a bit extreme when there is an easy alternative

[ndeloof]

!reset and !override by design are removed during the merge process are they are directives used during the merge process and changing the resulting yaml tree.

[ndeloof]

I'm closing this issue as "not planned". config command never has been designed to produce intermediate compose.yaml models to be used as a later override, and doing so would be highly challenging

[liamjones]

@jhrotko Yes, our real setup is much more complex and while base compose.yml changes frequently the components that get used in the override rarely change.

@ndeloof Thanks, I suspected that would be the answer but thought I'd check. I'm already working on an alternative solution.