[Examples] Add docker compose example to run multiple containers

[romilbhardwaj]

Simple example showing how to use docker compose to launch multiple containers on a SkyPilot cluster.

• sky launch -c myclus compose_example.yaml

[concretevitamin]

This is great to have @romilbhardwaj!

One issue with running it on Azure:

» sky launch compose_example.yaml -c dbg --cloud azure --cpus 2+ --down --gpus T4:4
...
(task, pid=26616)  gpu-app1 Pulled
(task, pid=26616)  gpu-app2 Pulled
(task, pid=26616)  Network sky_workdir_default  Creating
(task, pid=26616)  Network sky_workdir_default  Created
(task, pid=26616)  Container sky_workdir-gpu-app2-1  Creating
(task, pid=26616)  Container sky_workdir-gpu-app1-1  Creating
(task, pid=26616)  Container sky_workdir-gpu-app2-1  Created
(task, pid=26616)  Container sky_workdir-gpu-app1-1  Created
(task, pid=26616) Attaching to sky_workdir-gpu-app1-1, sky_workdir-gpu-app2-1
(task, pid=26616) Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=11.8, please update your driver to a newer version, or use an earlier cuda container: unknown
ERROR: Job 1 failed with return code list: [1]
INFO: Job finished (status: FAILED).
...

This may be related to our default GPU image on Azure having CUDA too outdated. Is it a quick fix?

[romilbhardwaj]

Ah good catch - we do need to update our azure image (#2751). For this PR, I've changed the version to 11.5.2 and tested it works on aws, az and gcp.

[concretevitamin]

I added -l 1 to here and L17 for nvidia-smi to loop forever.

It appears both containers print the same GPU ID. Note the SkyPilot task has 2 GPUs assigned, so GPUs 0 and 1 are available.

Is there any env var (CUDA_VISIBLE_DEVICES?) we can add to this file to show how to distribute the containers to GPUs 0 and 1 respectively? Can even be a comment.

[romilbhardwaj]

Good point - I've changed from count to explicit device_id. Note that nvidia-docker remaps device ids, so from within gpu-app2 container the GPU ID visible will be ['0'] (though it maps to physical device 1). Also added this as a comment.

[concretevitamin]

LGTM!