CI CD

CI/CD Workflow

This document describes the continuous integration and deployment pipeline.

Overview

┌─────────────────────────────────────────────────────────────────────┐
│                         Push / PR to main                           │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                        CI: Lint & Validate                          │
│  • Bash syntax check (bash -n)                                      │
│  • Source verification (makepkg --verifysource)                     │
│  • Detect changed packages                                          │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     CD Phase 1: AUR Publish                         │
│  • For packages with .aur marker                                    │
│  • Generate .SRCINFO                                                │
│  • Push to AUR git repos                                            │
│  • Must complete before Phase 2                                     │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     CD Phase 2: Local Build                         │
│  • For packages with .local marker                                  │
│  • Resolve deps via yay (repos + AUR)                               │
│  • Build with makepkg                                               │
│  • Run smoke tests (check.sh)                                       │
└─────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     CD Phase 3: Release                             │
│  • Sign packages with GPG                                           │
│  • Update repo database                                             │
│  • Upload to GitHub releases                                        │
└─────────────────────────────────────────────────────────────────────┘

Dependency Resolution

The yay Helper

CI uses yay to resolve package dependencies from multiple sources:

1. Official repositories (core, extra, multilib)
2. AUR (Arch User Repository)
3. Custom repos (this repo, if configured)

# setup.sh installs yay
su builder -c "cd /tmp && git clone https://aur.archlinux.org/yay-bin.git && cd yay-bin && makepkg -si --noconfirm"

# build.sh uses yay for dependency resolution
DEPS=$(makepkg --printsrcinfo | awk '/depends|makedepends/ {print $3}')
yay -S --noconfirm --asdeps --needed $DEPS

Why yay?

• Unified interface: Same command for repo and AUR packages
• Automatic AUR builds: Builds AUR deps from source automatically
• Dependency chains: Resolves transitive dependencies

Race Conditions

The Problem

When a package and its dependencies are both managed in this repo:

keeper-secrets-manager-cli (.local + .aur)
    └── depends: keeper-secrets-manager-helper (.aur)

If both are updated in the same commit:

1. CI detects both need publishing
2. If keeper-secrets-manager-cli builds first...
3. yay tries to fetch keeper-secrets-manager-helper from AUR
4. But the new version isn't pushed yet → build fails

Solution: Phased Deployment

Phase 1 (AUR Publish) must complete before Phase 2 (Local Build):

jobs:
  aur-publish:
    # Push all .aur packages to AUR first
    steps:
      - run: push_to_aur.sh

  local-build:
    needs: [aur-publish]  # Wait for AUR packages
    strategy:
      matrix:
        package: ${{ needs.prepare.outputs.local_packages }}
    steps:
      - run: build.sh ${{ matrix.package }}

This ensures:

1. AUR-only deps are available before local builds start
2. Packages with both markers get pushed to AUR, then built locally

Dependency Order Within Phases

For packages with local dependencies on other local packages (not via AUR):

Option A: Topological sort
  - Parse depends from PKGBUILDs
  - Build in dependency order
  - Complex but robust

Option B: Multi-stage builds
  - Stage 1: Packages with no local deps
  - Stage 2: Packages depending on Stage 1
  - Simpler but requires manual staging

Option C: Local repo bootstrap
  - Build all packages
  - Add built packages to local repo
  - Rebuild failures with local repo available
  - Self-healing but slower

Current implementation uses Option A with yay handling the complexity.

Version Tracking

Automated version checks run on schedule (.github/workflows/watch.yml):

# scripts/packages/<pkgname>.sh
check_pkgname() {
    # Fetch latest version from upstream
    local latest=$(curl -s "https://pypi.org/pypi/package/json" | jq -r .info.version)

    # Compare and update if needed
    perform_update "pkgname" "$latest"
}

Sources:

• npm: npm view @scope/package version
• PyPI: https://pypi.org/pypi/<package>/json
• GitHub: https://api.github.com/repos/org/repo/releases/latest

Secrets Required

Secret	Purpose
GPG_PRIVATE_KEY	Sign packages
GPG_PASSPHRASE	Unlock signing key
AUR_SSH_KEY	Push to AUR git repos
GITHUB_TOKEN	Upload release assets (auto-provided)

Manual Triggers

Force Rebuild All

Actions → "Build packages" → "Run workflow" → Check "Rebuild all packages"

Useful when:

• Dependency updated in repos
• Build environment changed
• Recovering from failed state

Fix Signatures

If packages exist but signatures are missing/invalid:

Actions → "Build packages" → "Run workflow" → Check "Fix signatures only"

GitHub Actions Gotchas

Artifact Naming

GitHub Actions artifact names cannot contain: " : < > | * ? \r \n \ /

Since packages are in pkgs/<name>/, the matrix value contains a slash. Use basename to extract just the package name:

- name: Get package name
  id: pkg
  run: echo "name=$(basename ${{ matrix.package }})" >> $GITHUB_OUTPUT

- uses: actions/upload-artifact@v4
  with:
    name: pkg-${{ steps.pkg.outputs.name }}  # Not pkg-${{ matrix.package }}

SSH in Containers

The archlinux:base-devel container doesn't include openssh. Install it in setup:

pacman -S --noconfirm --needed openssh

For AUR publishing, use GIT_SSH_COMMAND to ensure git uses the correct SSH config:

export GIT_SSH_COMMAND="ssh -i ~/.ssh/aur -o StrictHostKeyChecking=no"

Package Filenames with Colons

Packages with epochs have colons in filenames (e.g., gemini-cli-1:0.26.0-1-any.pkg.tar.zst). GitHub Actions cannot handle colons in artifact paths on some filesystems.

build.sh renames these before upload:

# Rename artifact if it contains colons (GitHub Actions compatibility)
for f in "$PKG"/*.pkg.tar.zst; do
    if [[ "$f" == *:* ]]; then
        new_name=$(echo "$f" | tr ':' '.')
        mv "$f" "$new_name"
    fi
done