Add Advanced Docker Documentation

docs/source/workflows/docker/advanced_guide_to_docker.md

@@ -0,0 +1,99 @@
+# Advanced Docker Topics
+
+## How Docker Works
+
+### The Linux Host (Usually a VM)
+
+Docker and its underlying container orchestration system fundamentally only runs on Linux. If you are using a Mac or
+Windows machine, your Docker system is running on a VM. On Windows this VM is usually Windows Subsystem for Linux (WSL)
+and on Mac this is usually the Docker Desktop-managed VM or a Colima VM.
+
+The container runtime and Docker system run on this VM host, which forwards the docker socket (`docker.sock`) through to
+your host system to allow you to use Docker commands in your host terminal. All the containers themselves are created on
+the VM filesystem.
+
+### The Container Runtime
+
+Most container orchestration systems run on `containerd`, a low level container orchestration layer that provides the
+foundation for other tools to manage and work with containers in a standard way. The containerd project was originally
+started by Docker but has been donated open source.
+
+### The CLI and Middle Layers
+
+Docker is one of many CLI tools that can interact with containerd to build and run containers. Another popular tool is
+`nerdctl`. Different tools have different levels of "middleware" that sit between the CLI and the containerd runtime.
+For example, Docker uses the open source Moby software as a middle layer between the Docker CLI and containerd to
+organize containers. Other tools interact with containerd directly.
+
+## Advanced Configuration
+
+### Running Containers Inside Containers
+
+It is common, especially when using [devcontainers](developing_in_devcontainers.md), that you might need to build
+and run a docker container inside another docker container.
+
+> [!CAUTION]
+> Neither DooD nor DinD is secure. DooD compromises the Linux host by allowing full filesystem access to the nested
+> containers and DinD compromises the host by elevating the container's system capabilities, allowing it to wreak havoc
+> on the Linux host.
+
+_See below for suggestions to isolate your host filesystem from your Linux VM._
+
+#### Docker Outside of Docker (DooD)
+
+In this model, all containers run on a Linux host with a docker daemon running. When containers are launched, the Linux
+host's docker.sock is mounted into the container, allowing the Docker CLI inside the container to interact with the
+host's Docker daemon.
+
+This means that all containers are sharing the same docker.sock and therefore the same docker context. For example, if
+you are running two containers and one builds a docker image, the other container can see that image!
+
+Pros:
+
+- Containers do not need any additional privileges to manage a docker daemon at the system level
+- Simple - just forward the docker.sock into the container
+
+Cons:
+
+- Container has control over the host Docker daemon
+- Poor isolation between running containers - docker builds can interfere because they share a daemon
+- Poor isolation from the VM host - nested containers can bind mount any directory on the Linux host because they are
+  using the host daemon and filesystem
+
+#### Docker In Docker (DinD)
+
+In this model, containers install a full Docker daemon internally. The container has its own docker daemon and is
+entirely segregated from the host's daemon when building nested containers. Typically DinD containers store their docker
+data in a Docker Volume to improve performance. This volume exists in the Linux Host's Docker system.
+
+Pros:
+
+- Isolation of Docker daemons - containers don't share the docker.sock
+- Isolation from Linux host filesystem - containers can't use nested containers to access host filesystem
+
+Cons:
+
+- Complicated - Each container comes with a Volume to store docker data and a full docker daemon running inside the
+  container
+- Elevated system permissions required - to manage the docker daemon, DinD containers require additional system level
+  permissions not usually granted to a container
+
+### Securing Your Data
+
+_This is always best practice but is essential if you are running docker in docker or docker outside of docker._
+
+Because Docker containers run only on the Linux host (usually VM), they only have access to the Linux filesystem. This
+is good but to make Docker more beginner friendly, most VM solutions (e.g. Docker Desktop and Colima) default to
+mounting your entire home directory into your Linux VM, giving your Docker containers access to all your files. If
+you've ever wondered how a Docker container isolated to a Linux VM can bind-mount a directory on your Mac, this is what
+makes that work seamlessly. In fact, the VM defaults to mounting your home directory to exactly the same path, so while
+it _seems_ like your bind-mount commands reflect your host filesystem, they really reflect the path inside the VM.
+
+```
+Mac Filesystem -> Linux VM -> Docker Container
+```
+
+Don't accept this default! Choose exactly what data your Docker containers should be able to access by limiting what
+your Linux VM is allowed to mount. Remember that the bind-mount path you give to your containers is really the path to
+the data _inside the Linux VM_.
+

docs/source/workflows/docker/beginner_guide_to_docker.md

@@ -157,10 +157,15 @@ docker run --platform linux/amd64 -it --name <container name> <image name>:lates
 docker login <registry_hostname>
 
 # View docker images
-docker images
+docker images # legacy
+docker image ls # modern
 
 # View docker containers
-docker ps -a
+docker ps -a # legacy
+docker container ls -a # modern
+
+# View docker volumes
+docker volume ls
 
 # Remove stopped containers
 docker container prune

docs/source/workflows/docker/developing_in_devcontainers.md

@@ -0,0 +1,56 @@
+# Developing in Devcontainers
+
+A devcontainer is a Docker container that contains your code and development environment. Most modern IDEs support
+connecting to a devcontainer for development. This allows you to develop against a stable environment that is not
+dependent on the configuration of the rest of your host system.
+
+Devcontainers are extremely flexible and can be configured in many ways via `devcontainer.json`, from running in GitHub
+Codespaces to bind-mounting your repo directory, to cloning a repo into a dedicated Docker volume.
+
+## Bind-Mounting Code
+
+You can configure your devcontainer to simply bind-mount a directory from the Linux VM (usually mounted into the VM
+from your host filesystem). This is the simplest way to work on code in a devcontainer but it comes with some downsides.
+
+Pros:
+
+- Dead simple. Just configure the devcontainer to bind-mount the directory.
+
+Cons:
+
+- Requires you to first mount the directory into your Linux VM running Docker (see [Advanced Docker
+  Configuration](advanced_guide_to_docker.md) for more details on why this is a bad idea).
+- Bind-mounts are slow because file system changes are propagated through layers of OS translation from container, to
+  VM, to host.
+
+
+## Volume-Mounting Code
+
+Alternatively, you can store your repo code (or really anything you want to work on) inside a Docker volume that lives
+entirely inside the Linux VM. Some IDEs (e.g. VSCode) actually provide a UI for this workflow, which clones a remote
+repository into a volume and then mounts that volume into a devcontainer. If the repo you clone contains a
+`.devcontainer/devcontainer.json` file, it uses that configuration. If the repo does not come with a devcontainer
+configuration, VSCode will prompt you to create one.
+
+Pros:
+
+- Fast! Volumes do not require OS filesystem layer translation so everything will feel snappier
+- Better isolation from host. You can avoid mounting your host filesystem into your Linux VM.
+
+Cons:
+
+- Ephemeral. If you delete your docker volume, your repo is gone. This may not be a problem if you push everything to a
+  git remote but any .gitignored files will be gone.
+- Less flexible. You don't get free access to your host filesystem. You only get your repo.
+
+
+## SSH and GPG Keys
+
+In pretty much any dev environment, you need access to SSH keys and GPG keys for authenticating with repositories
+and signing commits. But usually, these keys live on your host machine and it's bad practice to be copying
+keys around.
+
+Luckily, in much the same way that Docker forwards `docker.sock` through to your host system, you can forward your
+SSH auth socket into your Linux VM as well as your GPG agent socket. In Colima this is as simple as setting
+`forwardAgent: true` in your `colima.yaml` configuration. VSCode will automatically pick these up and provide
+access to your SSH keys and GPG keys inside your devcontainer.

docs/source/workflows/docker/index.rst

@@ -5,6 +5,8 @@ Docker
     :maxdepth: 1
 
     beginner_guide_to_docker
+    advanced_guide_to_docker
+    developing_in_devcontainers
     containerizing_idl_with_docker
     docker_compose_examples
     export_display_with_docker