cicd pipeline

CI/CD Pipelines & Orchestration

This module covers the automation scripts, GitHub Action workflow definitions, and cross-registry image synchronization tooling of the LabNow Docker ecosystem.

---

1. Orchestration Script (tool.sh)

Located at the repository root, tool.sh coordinates local and automated builds. It configures registries, tags, and helper functions.

Namespace & Branch-Based Tagging Style

The script determines the registry namespace and tags dynamically based on the current Git branch:

• CI_PROJECT_BRANCH: Defaults to $GITHUB_HEAD_REF or fallback to "main".
• NAMESPACE_SUFFIX:
  • If the branch is main, the suffix is empty: "".
  • Otherwise, it is formatted as: "0" + {first segment of branch before "/"} (e.g. feat/add-login -> suffix is 0feat).
• CI_PROJECT_NAMESPACE: Concatenates project namespace with the branch suffix (e.g. LabNow-ai -> LabNow-ai0feat).
• TAG_SUFFIX: Short hash suffix -$(git rev-parse --short HEAD).

Core Build Functions

• build_image <img_name> <tag> <dockerfile_path> [build-args]: Builds the image using --compress --force-rm=true. It tags the output as ${REGISTRY_DST}/${IMG_NAMESPACE}/${img_name}:${tag} and creates a timestamped version Y.m.d.HM${TAG_SUFFIX}.
• build_image_no_tag: Builds the image without adding the timestamped version tag (typically used for intermediate build stages).
• alias_image <img1> <tag1> <img2> <tag2>: Retags a compiled local image to another repository name or alias.
• push_image [keyword]: Logs into the destination registry via stdin, selects local images matching [keyword], and pushes them.
• clear_images: Clears historical local images to prevent Docker daemon disk starvation.

---

2. GitHub Actions Workflows

Automated builds are split into two workflows located under .github/workflows/:

A. Core Workflow (build-docker.yml)

• Trigger: Pushes or PRs targeting main branch (ignoring changes solely in markdown files).
• Jobs:
  • job-base: Compiles the core atom and base images.
  • job-python/job-core/job-py-std/job-node/job-rust/job-jdk etc.: Depend on job-base and run concurrently to build their specific language environments.
  • job-docker_kit: Builds the docker-kit image containing image-syncer.
  • sync_images: Runs final registry migrations once dependencies are ready.

B. GPU Workflow (build-docker-gpu.yml)

• Trigger: Same trigger conditions.
• Jobs:
  • job-cuda_128 / job-cuda_126: Compiles CUDA-specific atom wrappers, builds conda-base wrappers, and builds final cuda base images.
  • job-torch_cuda128 / job-tf2 / job-paddle_cuda126 etc.: Depend on the respective CUDA jobs and compile python profiles for Deep Learning stacks.
  • job-core-cuda: Builds the full stack CUDA environment.

---

3. Registry Mirroring & Syncing (docker-kit)

Built using docker_docker_kit/docker-kit.Dockerfile, the docker-kit image packages the tools required to mirror built images across registries.

Key Tools in docker-kit

• yq: Installed via setup_yq for YAML files processing.
• Docker Compose: Sourced from script-setup-docker.sh -> setup_docker_compose.
• image-syncer: Sourced from setup_docker_syncer and mapped to /opt/utils/image-syncer/.

Synchronizer Scripts (image-syncer/)

• run_jobs.py: A python script that reads registry authentication credentials from auth.json, parses workflow files to identify target images, and starts mirroring processes.
• run_sync.py: Executes parallel image-syncer shell invocations to migrate images from the primary destination registry to secondary mirror registries (e.g. Quay.io to regional mirrors).