Docker Compose V2 and related build fixes (#1335)

Misc build fixes discussed in b/245490832

1. Replacing multiple YAML '<<:' keys with array
2. Replacing 'python' with 'python3'
3. Removing duplicate 'image:' key in xvfb (low confidence, I don't have
access to test it)
4. Replacing 'docker-compose' commands with 'docker compose' (with
space)

b/245490832

.github/actions/docker/action.yaml

@@ -79,7 +79,7 @@ runs:
           echo "need_to_build=false" >> $GITHUB_ENV
         fi
       shell: bash
-    - name: Build containers with docker-compose
+    - name: Build containers with Docker Compose
       id: build-image
       if: env.need_to_build == 'true'
       env:

.github/actions/docker_win/action.yaml

@@ -56,7 +56,7 @@ runs:
           echo "need_to_build=false" >> $GITHUB_ENV
         fi
       shell: bash
-    - name: Build containers with docker-compose
+    - name: Build containers with Docker Compose
       if: env.need_to_build == 'true'
       env:
         DOCKER_CPUS: 2

AUTHORS

@@ -17,3 +17,4 @@ NVIDIA <*@nvidia.com>
 MediaTek <*@mediatek.com>
 MIPS <*@mips.com>
 Vewd <*@vewd.com>
+Broadcom <*@broadcom.com>

CONTRIBUTING.md

@@ -89,11 +89,11 @@ You can find documentation about it at https://pre-commit.com/.
 
 #### Running unit tests
 
-First, ensure Docker and docker-compose are installed on your system. Then,
+First, ensure Docker and Docker Compose are installed on your system. Then,
 you can run unit tests for our linux reference implementation using:
 
 ```bash
-$ docker-compose up --build --no-start linux-x64x11-unittest
-$ PLATFORM=linux-x64x11 CONFIG=devel TARGET=all docker-compose run linux-x64x11
-$ PLATFORM=linux-x64x11 CONFIG=devel docker-compose run linux-x64x11-unittest
+$ docker compose up --build --no-start linux-x64x11-unittest
+$ PLATFORM=linux-x64x11 CONFIG=devel TARGET=all docker compose run linux-x64x11
+$ PLATFORM=linux-x64x11 CONFIG=devel docker compose run linux-x64x11-unittest
 ```

cobalt/doc/docker_build.md

@@ -5,18 +5,18 @@ definitions for simplifying managing build environments.
 
 The instructions below assume Docker is installed and is able to run basic
 [`hello-world` verification](https://docs.docker.com/get-started/#test-docker-installation).
-`docker-compose` command is expected to be available as well.
+`docker compose` command is expected to be available as well.
 
 ## Usage
 
 The simplest usage is:
 
-  `docker-compose run <platform>`
+  `docker compose run <platform>`
 
 By default, a debug build will be built, with `cobalt` as a target.
 You can override this with an environment variable, e.g.
 
-  `docker-compose run -e CONFIG=devel -e TARGET=nplb <platform>`
+  `docker compose run -e CONFIG=devel -e TARGET=nplb <platform>`
 
 where config is one of the four optimization levels, debug, devel, qa and gold,
 and target is the build target passed to `ninja`
@@ -32,9 +32,9 @@ files in `src/out/<platform>` directory have `root` as file owner.
 ## Customization
 
 To parametrize base operating system images used for the build, pass BASE_OS
-as an argument to docker-compose as follows:
+as an argument to docker compose as follows:
 
-  `docker-compose build --build-arg BASE_OS="ubuntu:bionic" base`
+  `docker compose build --build-arg BASE_OS="ubuntu:bionic" base`
 
 This parameter is defined in `docker/linux/base/Dockerfile` and is passed to
 Docker `FROM ...` statement.
@@ -73,6 +73,6 @@ docker pull ghcr.io/youtube/cobalt/cobalt-build-evergreen:23.lts
 To debug build issues, enter the shell of the corresponding build container by
 launching the bash shell, i.e.
 
-  `docker-compose run linux-x64x11 /bin/bash`
+  `docker compose run linux-x64x11 /bin/bash`
 
 and try to build cobalt [with the usual `gn / ninja` flow](../../README.md#building-and-running-the-code).

cobalt/site/docs/development/setup-docker.md

@@ -7,7 +7,7 @@ We provide <a
 href="https://github.com/youtube/cobalt/tree/main/docker/linux/">Docker image definitions</a> to simplify managing build environments.
 
 The instructions below assume Docker is installed and is able to run basic
-hello-world verification. `docker-compose` command is expected to be available as well.
+hello-world verification. `docker compose` command is expected to be available as well.
 
 ## Set up your workstation
 
@@ -24,14 +24,14 @@ $ cd cobalt
 The simplest usage is:
 
 ```
-docker-compose run <platform>
+docker compose run <platform>
 ```
 
 By default, a `debug` build will be built, with `cobalt` as a target.
 You can override this with an environment variable, e.g.
 
 ```
-docker-compose run -e CONFIG=devel -e TARGET=nplb <platform>
+docker compose run -e CONFIG=devel -e TARGET=nplb <platform>
 ```
 
 where config is one of the four optimization levels, `debug`, `devel`,
@@ -53,16 +53,16 @@ We have a separate docker compose file for windows. Use the -f or --file flags
 to specify a configuration file to use.
 
 ```
-docker-compose -f docker-compose-windows.yml run win-win32
+docker compose -f docker-compose-windows.yml run win-win32
 ```
 
 ### Customization
 
 To parametrize base operating system images used for the build, pass
-`BASE_OS` as an argument to `docker-compose` as follows:
+`BASE_OS` as an argument to `docker compose` as follows:
 
 ```
-docker-compose build --build-arg BASE_OS="ubuntu:bionic" base
+docker compose build --build-arg BASE_OS="ubuntu:bionic" base
 ```
 
 This parameter is defined in `docker/linux/base/Dockerfile` and is passed
@@ -95,7 +95,7 @@ To debug build issues, enter the shell of the corresponding build container
 by launching the bash shell, i.e.
 
 ```
-docker-compose run linux-x64x11 /bin/bash
+docker compose run linux-x64x11 /bin/bash
 ```
 
 and try to build Cobalt with the <a

cobalt/site/docs/gen/cobalt/doc/docker_build.md

@@ -9,18 +9,18 @@ definitions for simplifying managing build environments.
 
 The instructions below assume Docker is installed and is able to run basic
 [`hello-world` verification](https://docs.docker.com/get-started/#test-docker-installation).
-`docker-compose` command is expected to be available as well.
+`docker compose` command is expected to be available as well.
 
 ## Usage
 
 The simplest usage is:
 
-  `docker-compose run <platform>`
+  `docker compose run <platform>`
 
 By default, a debug build will be built, with `cobalt` as a target.
 You can override this with an environment variable, e.g.
 
-  `docker-compose run -e CONFIG=devel -e TARGET=nplb <platform>`
+  `docker compose run -e CONFIG=devel -e TARGET=nplb <platform>`
 
 where config is one of the four optimization levels, debug, devel, qa and gold,
 and target is the build target passed to `ninja`
@@ -36,9 +36,9 @@ files in `src/out/<platform>` directory have `root` as file owner.
 ## Customization
 
 To parametrize base operating system images used for the build, pass BASE_OS
-as an argument to docker-compose as follows:
+as an argument to docker compose as follows:
 
-  `docker-compose build --build-arg BASE_OS="ubuntu:bionic" base`
+  `docker compose build --build-arg BASE_OS="ubuntu:bionic" base`
 
 This parameter is defined in `docker/linux/base/Dockerfile` and is passed to
 Docker `FROM ...` statement.
@@ -77,6 +77,6 @@ docker pull ghcr.io/youtube/cobalt/cobalt-build-evergreen:23.lts
 To debug build issues, enter the shell of the corresponding build container by
 launching the bash shell, i.e.
 
-  `docker-compose run linux-x64x11 /bin/bash`
+  `docker compose run linux-x64x11 /bin/bash`
 
 and try to build cobalt [with the usual `gn / ninja` flow](../../README.md#building-and-running-the-code).

docker-compose-windows-internal.yml

@@ -13,7 +13,7 @@
 # limitations under the License.
 
 # This file should be combined with docker-compose-windows.yml by specifying
-# docker-compose -f docker-compose-windows.yml -f docker-compose-windows-internal.yml ...
+# docker compose -f docker-compose-windows.yml -f docker-compose-windows-internal.yml ...
 # on the command line.
 
 version: '2.4'

docker-compose.yml

@@ -29,8 +29,9 @@ x-build-volumes: &build-volumes
     - ${CCACHE_DIR:-container-ccache}:/root/ccache
 
 x-build-common-definitions: &build-common-definitions
-  <<: *common-definitions
-  <<: *build-volumes
+  <<:
+    - *common-definitions
+    - *build-volumes
   depends_on:
     - build-base
 
@@ -200,8 +201,9 @@ services:
       TARGET: ${TARGET:-cobalt_install}
 
   linux-x64x11-bionic:
-    <<: *common-definitions
-    <<: *build-volumes
+    <<:
+      - *common-definitions
+      - *build-volumes
     build:
       context: ./docker/linux
       dockerfile: linux-x64x11/Dockerfile
@@ -213,8 +215,9 @@ services:
     scale: 0
 
   linux-x64x11-gcc:
-    <<: *common-definitions
-    <<: *build-volumes
+    <<:
+      - *common-definitions
+      - *build-volumes
     build:
       context: ./docker/linux
       dockerfile: gcc-6-3/Dockerfile
@@ -227,8 +230,9 @@ services:
       - linux-x64x11-bionic
 
   build-linux-gcc:
-    <<: *common-definitions
-    <<: *build-volumes
+    <<:
+      - *common-definitions
+      - *build-volumes
     build:
       context: ./docker/linux
       dockerfile: gcc-6-3/Dockerfile
@@ -237,8 +241,9 @@ services:
       - linux-x64x11-bionic
 
   linux-x64x11-clang-3-9:
-    <<: *common-definitions
-    <<: *build-volumes
+    <<:
+      - *common-definitions
+      - *build-volumes
     build:
       context: ./docker/linux/
       dockerfile: clang-3-9/Dockerfile
@@ -251,8 +256,9 @@ services:
       - linux-x64x11-bionic
 
   build-linux-clang-3-9:
-    <<: *common-definitions
-    <<: *build-volumes
+    <<:
+      - *common-definitions
+      - *build-volumes
     build:
       context: ./docker/linux/
       dockerfile: clang-3-9/Dockerfile
@@ -455,11 +461,11 @@ services:
 
   # Example usage of unittest:
   # 1. Build the containers for which you want to unittest
-  # docker-compose up --build --no-start linux-x64x11-unittest
+  # docker compose up --build --no-start linux-x64x11-unittest
   # 2. Build the 'all' target for the platform you want to test
-  # PLATFORM=linux-x64x11 CONFIG=devel TARGET=all docker-compose run linux-x64x11
+  # PLATFORM=linux-x64x11 CONFIG=devel TARGET=all docker compose run linux-x64x11
   # 3. Run the unittests for that target.
-  # PLATFORM=linux-x64x11 CONFIG=devel docker-compose run linux-x64x11-unittest
+  # PLATFORM=linux-x64x11 CONFIG=devel docker compose run linux-x64x11-unittest
   linux-x64x11-unittest:
     <<: *shared-unittest-definitions
 
@@ -490,10 +496,9 @@ services:
     entrypoint: ["python3", "/unittest_docker_launcher.py", "3"]
 
   xvfb:
-    image: xvfb
+    image: cobalt-xvfb
     ports:
       - "99:99"
-    image: cobalt-xvfb
     build:
       context: ./docker/linux/
       dockerfile: xvfb/Dockerfile

docker/docsite/README.md

@@ -12,6 +12,6 @@ output the site content in the directory `out/deploy/www` at the root of the
 Cobalt directory, and copy the app.yaml deployable into `out/deploy`.
 
 # How To Run
-docker-compose build --no-cache --build-arg USER=defaultuser --build-arg UID=$(id -u) --build-arg GID=$(id -g) SERVICE
+docker compose build --no-cache --build-arg USER=defaultuser --build-arg UID=$(id -u) --build-arg GID=$(id -g) SERVICE
 
-docker-compose up SERVICE
+docker compose up SERVICE

docker/linux/base/Dockerfile

@@ -33,4 +33,4 @@ RUN apt update -qqy \
         curl xz-utils \
     && /opt/clean-after-apt.sh
 
-CMD ["/usr/bin/python","--version"]
+CMD ["/usr/bin/python3","--version"]

docker/linux/base/build/Dockerfile

@@ -114,4 +114,4 @@ RUN cd /tmp \
 RUN git config --global --add safe.directory /code
 
 WORKDIR /code
-CMD ["/usr/bin/python","--version"]
+CMD ["/usr/bin/python3","--version"]

docker/linux/unittest/unittest_docker_launcher.py

@@ -48,7 +48,7 @@ def main(argv):
   env_platform = os.getenv('PLATFORM')
   env_config = os.getenv('CONFIG')
   test_command = [
-      'python', '/app_launcher_out/starboard/tools/testing/test_runner.py',
+      'python3', '/app_launcher_out/starboard/tools/testing/test_runner.py',
       '--run', '-o', out_dir, '-p', env_platform, '-c', env_config, '-l'
   ]