Bundling containers for a kernel session

[achimnol]

For large-scale computations, sometimes we need to run multiple containers on different hosts for resource aggregation and distributed/parallel processing.

In the past, this was very difficult to implement because Docker's networking was limited to mounting a container to another via an hostname alias (--link), which is essentially one-to-one private links. Now, it's 2017, and Docker offers a nice distributed coordination called "Swarm" which includes overlay networking.

Docker Swarm uses the Raft algorithm to share node information and any new Docker daemon can join to an existing Swarm via host:port and a secret token. Once joined, any containers of the daemons in the swarm can be connected to volatile overlay networks created and destroyed at runtime.

Let's try this and support multi-container distributed computing!

Update for 2020!

Docker Swarm has problems with overlapped IP addresses in different overlay networks and creating/destroying and attaching/detaching networks is proven to be unstable.

• https://success.docker.com/article/overlay-networks-with-overlapping-ip-addresses
• https://success.docker.com/article/force-remove-task-from-network

After some testing by @kyujin-cho , we decided to fall back to the "classic" Swarm mode, which uses an external etcd to manage multi-host networks, and use namespaced container hostname aliasing to access other containers in the same overlay network.

Basically we keep the same "kernels" table extension as we prototyped in 2017-2018. A single record of the kernels table correspond to a container and multiple records may share the same sess_id indicating that they belong to a overlay cluster.

Phase 1

• Automatic reconfiguration of existing Docker daemons in agents so that they use our etcd.
• Add a simple integration test script that manually spawns two containers in different hosts to check if the overlay network actually works. Let's place the script into the scripts directory of this meta repository.

Phase 2

• Design and implement a template format for a compute session. → Task template #70
  • A task template contains the target image, resource occupation, environment variables, default vfolder mounts, bootstrap script, etc.
• Design and implement a template format for a cluster composed of session templates.
  • The cluster template may declare multiple "roles" of containers and the min/max numbers of containers for each role.
  • Each container should have special environment variables so that they can detect the role and index. (e.g., BACKEND_CLUSTER_ROLE, BACKEND_CLUSTER_ROLE_IDX)
• (Re-)implement lifecycle mgmt. of multiple containers for a single session
  • First, use a plain for-loop to trigger creation/destruction of each kernel.
  • Set up custom hostnames via Docker daemon/API so that all containers in the same overlay cluster can access each other.
    • e.g., Spawning 2 "master" and 12 "worker" nodes: "master1", "master2", "worker1", "worker2", ..., "worker12"
    • Don't pad zeros for lexicographical ordering beacuse it causes extra complexity in automated scripts for images and task templates.
  • We need to extend the semantics of kernel lifecycle operations:
    • All lifecycle operations against a session must wait until the same operations applied to all kernels of the same session to complete.
    • Restarting and terminating any of the kernels triggers the same operation of the whole overlay cluster.
  • The kernel creation options like environment variables, vfolder mounts must be applied to all kernels of a session and override those defined by task templates which also override those defined in the images.
• Extend the GUI to have "collapsable" kernel list so that kernels for a single session are grouped and folded together.
  • The primary kernel of a session (which is exposed in the main session list) is the first indexed kernel of the first role as in the defined order.
  • Each kernel with different roles may define their own service ports and those service ports are transparently supported.

Phase 3

• Keep consistency when the manager is interrupted (e.g., restarted) during multi-container spawning and destruction (i.e., partially provisioned).
  • We may need to add a column specifying "desired state" so that we could resume and continue the cluster provisioning jobs upon manager restarts.
• Optimize the provisioning using asyncio.gather() with proper interruption handling.

┆Issue is synchronized with this Asana task by Unito

[achimnol]

Now the server-side implementation is done with the v20.09.0 release, with the following work:

• Improve multi-node multi-container scheduling backend.ai-manager#372
• Multi-node multi-container stabilization backend.ai-manager#373
• Fix races of kernel creation events backend.ai-manager#374
• Stabilization of multi-node multi-container sessions (Phase 3) backend.ai-manager#376
• lablup/backend.ai-agent#254
• lablup/backend.ai-agent#255
• lablup/backend.ai-agent#256
• lablup/backend.ai-agent#257

[achimnol]

Closing as completed and let's handle fine-grained UI improvements in separate issues.