[BUG] On V2.24.0 a Circular reference Error

[CodeZHXS]

Description

When I use Docker Compose standalone v2.24.0, I encountered a bug. If you follow the steps below, you should be able to reproduce it:

1. Pull some images:

docker pull yeasy/hyperledger-fabric-base:0.6-dp \
  && docker pull yeasy/hyperledger-fabric-peer:0.6-dp \
  && docker pull yeasy/hyperledger-fabric-membersrvc:0.6-dp \
  && docker pull yeasy/blockchain-explorer:latest \
  && docker tag yeasy/hyperledger-fabric-peer:0.6-dp hyperledger/fabric-peer \
  && docker tag yeasy/hyperledger-fabric-base:0.6-dp hyperledger/fabric-baseimage \
  && docker tag yeasy/hyperledger-fabric-membersrvc:0.6-dp hyperledger/fabric-membersrvc

2. Create $2$ new YAML files，where 4-peers.yml depends on peer.yml.

Steps To Reproduce

When I use Docker Compose standalone v2.24.0, I encountered a bug. If you follow the steps below, you should be able to reproduce it:

1. Pull some images:

docker pull yeasy/hyperledger-fabric-base:0.6-dp \
  && docker pull yeasy/hyperledger-fabric-peer:0.6-dp \
  && docker pull yeasy/hyperledger-fabric-membersrvc:0.6-dp \
  && docker pull yeasy/blockchain-explorer:latest \
  && docker tag yeasy/hyperledger-fabric-peer:0.6-dp hyperledger/fabric-peer \
  && docker tag yeasy/hyperledger-fabric-base:0.6-dp hyperledger/fabric-baseimage \
  && docker tag yeasy/hyperledger-fabric-membersrvc:0.6-dp hyperledger/fabric-membersrvc

2. Create $2$ new YAML files, where 4-peers.yml depends on peer.yml.
   The peer.yml is:

# This is the default base file to config env and command
# Notice that chaincode is executed inside docker in default net mode
# https://github.com/yeasy/docker-compose-files

# Depends on the hyperledger/fabric-peer image.

# This utilized the pbft as consensus
# See https://github.com/hyperledger/fabric/blob/master/docs/dev-setup/devnet-setup.md#using-consensus-plugin for more details.

version: '2'

services:
  vp:  # vp node no needing membersrvc
    image: yeasy/hyperledger-fabric-peer:0.6-dp
   #image: yeasy/hyperledger-fabric-peer:0.6-dp
   #image: hyperledger/fabric-peer:x86_64-0.6.1-preview
   #image: ibmblockchain/fabric-peer:x86_64-0.6.1-preview
    restart: unless-stopped
    environment:
      - CORE_PEER_ADDRESSAUTODETECT=true
      - CORE_PEER_NETWORKID=dev
      - CORE_LOGGING_LEVEL=info  #critical, error, warning, notice, info, debug
     #- CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=fabric_pbft  # uncomment this to use specific network
      # The following section enables pbft consensus
      - CORE_PEER_VALIDATOR_CONSENSUS_PLUGIN=pbft  # noops, pbft
      - CORE_PBFT_GENERAL_MODE=batch  # batch. classic and sieve will be not supported again.
      - CORE_PBFT_GENERAL_N=4
      - CORE_PBFT_GENERAL_BATCHSIZE=2  # only useful when in batch mode
      - CORE_PBFT_GENERAL_TIMEOUT_REQUEST=2s
    expose:
      - "7050"  # Rest
      - "7051"  # Grpc
      - "7052"  # Peer CLI
      - "7053"  # Peer Event
      - "7054"  # eCAP
      - "7055"  # eCAA
      - "7056"  # tCAP
      - "7057"  # eCAA
      - "7058"  # tlsCAP
      - "7059"  # tlsCAA
    volumes: # docker.sock is mapped as the default CORE_VM_ENDPOINT
      - /var/run/docker.sock:/var/run/docker.sock
    command: peer node start  # this repeats the default cmd to make sure
  # command: sh -c "sleep 5; peer node start"  # peer node should have booting delay with CA

  vp_sec: # vp node needing membersrvc suppot
    extends:
      service: vp
    environment:
      - CORE_SECURITY_ENABLED=true
      - CORE_PEER_PKI_ECA_PADDR=membersrvc0:7054
      - CORE_PEER_PKI_TCA_PADDR=membersrvc0:7054
      - CORE_PEER_PKI_TLSCA_PADDR=membersrvc0:7054

---

The 4-peers.yml is:

# This compose file will start a cluster with 4 hyperledger peer nodes
#  vp0: validating node as root
#  vp1: validating node as peer
#  vp2: validating node as peer
#  vp3: validating node as peer
# https://github.com/yeasy/docker-compose-files

version: '2'

services:
  # validating node as the root
  # vp0 will also be used for client interactive operations
  # If you want to run fabric command on the host, then map 7051:7051 to host
  # port, or use like `CORE_PEER_ADDRESS=172.17.0.2:7051` to specify peer addr.
  vp0:
    extends:
      file: peer.yml
      service: vp
    hostname: vp0
    environment:
      - CORE_PEER_ID=vp0
    ports:
      - "7050:7050"
      #- "7051:7051"

  # validating node
  vp1:
    extends:
      file: peer.yml
      service: vp
    hostname: vp1
    environment:
      - CORE_PEER_ID=vp1
      - CORE_PEER_DISCOVERY_ROOTNODE=vp0:7051
    links:
      - vp0

  # validating node
  vp2:
    extends:
      file: peer.yml
      service: vp
    hostname: vp2
    environment:
      - CORE_PEER_ID=vp2
      - CORE_PEER_DISCOVERY_ROOTNODE=vp0:7051
    links:
      - vp0


  # validating node
  vp3:
    extends:
      file: peer.yml
      service: vp
    hostname: vp3
    environment:
      - CORE_PEER_ID=vp3
      - CORE_PEER_DISCOVERY_ROOTNODE=vp0:7051
    links:
      - vp0

#networks:
#  default:
#    external:
#      name: fabric_pbft

3. run docker-compose -f 4-peers.yml up, then you will get error like:

 Circular reference:
 vp0 in /root/pbft/4-peers.yml
 extends vp_sec in /root/pbft/peer.yml
 extends vp1 in /root/pbft/4-peers.yml
 extends vp_sec in /root/pbft/peer.yml

I have confirmed that there is no issue when using docker-compose v2.23.3.

Compose Version

docker compose version
Docker Compose version v2.21.0

docker-compose version
Docker Compose version v2.24.0

Docker Environment

docker info
Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dd1e886e55dd695541fdcd67420c2888645a495
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-91-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.786GiB
 Name: linux2
 ID: 62466d6a-e2a2-4a89-871e-a6f96786af06
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://hub-mirror.c.163.com/
  https://docker.mirrors.ustc.edu.cn/
 Live Restore Enabled: false


root@linux2 ~/pbft
❯ systemctl daemon-reload
systemctl restart docker


root@linux2 ~/pbft 7s
❯ docker info
Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dd1e886e55dd695541fdcd67420c2888645a495
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-91-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.786GiB
 Name: linux2
 ID: 62466d6a-e2a2-4a89-871e-a6f96786af06
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

[dkadioglu]

The same happens for me, as I already described in the following issue at the Zabbix Docker repo: zabbix/zabbix-docker#1121
With 2.23.3 the same compose files run without error.

[ndeloof]

I can reproduce this issue, will investigate this today and include a test-case with your example to ensure we don't get such a regression in the future

[mauke]

Just ran into this issue. Just for completeness, a reduced test case:

File a:

name: tmp

services:
  s0:
    extends:
      file: b
      service: s-base

  s1:
    extends:
      file: b
      service: s-base

File b:

services:
  s-base:
    image: busybox

  s-other:
    extends: s-base

Then run:

$ docker run -v ".:/mnt:ro" --rm docker:24.0.7-cli docker compose -f /mnt/a config
Circular reference:
  s1 in /mnt/a
  extends s-other in /mnt/b
  extends s0 in /mnt/a
  extends s-other in /mnt/b

("extends s-other" is of course wrong; the entry isn't actually used by any service.)