Gen3 Admin Self Hosted Updating

Self-Hosted Updating

Start Here

1. Run commands on the same host that installed the cluster (or any host with /var/lib/gt-ai-os/admin state and cluster kubeconfig access).
2. Know your namespace (default gt-ai-os-prod). Pin the target with TO_VERSION=v3.0.5-stable (or your tag) or resolve latest from GitHub releases.
3. Use gt-ai-os-admin update to upgrade the application in the existing namespace. This does not reinstall RKE2.
4. After upgrade, run gt-ai-os-admin validate and sign in to Control Panel and the tenant app.
5. For interactive menus, use manage-ai-os.sh upgrade or see Advanced Management.

Why this matters

The Control Panel Updates workspace may show an under-development placeholder in interim builds. Host-level upgrades today use gt-ai-os-admin on the install server. This runbook is the supported path until in-control-panel update orchestration ships.

For v3.0.2+ public releases (GT-Edge-AI/GT-AI-OS, ghcr.io/gt-edge-ai), you usually do not need a GitHub PAT when release assets and registry images are publicly reachable.

Details

List published releases

sudo env GT_AI_OS_ADMIN_CONFIG_DIR=/var/lib/gt-ai-os/admin \
  KUBECONFIG=/etc/rancher/rke2/rke2.yaml \
  PATH="/var/lib/rancher/rke2/bin:/usr/local/bin:$PATH" \
  gt-ai-os-admin release list --limit 20

Shared operator state lives under /var/lib/gt-ai-os/admin (state.json, generated manifests, and stored Cloudflare profiles when you use a Cloudflare install).

Non-interactive upgrade (automation)

Step 1 refreshes operator scripts and installs the matching gt-ai-os-admin for TO_VERSION. Steps 2–4 upgrade, validate, and report.

export NAMESPACE="gt-ai-os-prod"
export TO_VERSION="$(curl -fsSL https://api.github.com/repos/GT-Edge-AI/GT-AI-OS/releases/latest | grep '"tag_name"' | head -1 | cut -d'"' -f4)"

sudo env GT_AI_OS_ADMIN_CONFIG_DIR=/var/lib/gt-ai-os/admin \
  KUBECONFIG=/etc/rancher/rke2/rke2.yaml \
  PATH="/var/lib/rancher/rke2/bin:/usr/local/bin:$PATH" \
  bash -c 'source /var/lib/gt-ai-os/operator-scripts/gt-ai-os-admin-env.sh && gt_ai_os_ensure_admin_cli_version "'"${TO_VERSION}"'"'

sudo env GT_AI_OS_ADMIN_CONFIG_DIR=/var/lib/gt-ai-os/admin \
  KUBECONFIG=/etc/rancher/rke2/rke2.yaml \
  PATH="/var/lib/rancher/rke2/bin:/usr/local/bin:$PATH" \
  gt-ai-os-admin update --yes --namespace "${NAMESPACE}" --to "${TO_VERSION}"

sudo env GT_AI_OS_ADMIN_CONFIG_DIR=/var/lib/gt-ai-os/admin \
  KUBECONFIG=/etc/rancher/rke2/rke2.yaml \
  PATH="/var/lib/rancher/rke2/bin:/usr/local/bin:$PATH" \
  gt-ai-os-admin validate --namespace "${NAMESPACE}"

sudo env GT_AI_OS_ADMIN_CONFIG_DIR=/var/lib/gt-ai-os/admin \
  KUBECONFIG=/etc/rancher/rke2/rke2.yaml \
  PATH="/var/lib/rancher/rke2/bin:/usr/local/bin:$PATH" \
  gt-ai-os-admin report --namespace "${NAMESPACE}"

--yes is required when stdin is not a TTY.

Interactive upgrade

sudo bash /var/lib/gt-ai-os/operator-scripts/manage-ai-os.sh upgrade

Or install the admin CLI for the target tag, then run gt-ai-os-admin update without --yes to confirm namespace and version at prompts. See Advanced Management for k9s, validate, and bootstrap-creds.

What update does

gt-ai-os-admin update loads stored instance state, fetches release assets for --to, refreshes Helm values and manifests, runs deploy validation and smoke checks (unless disabled), and saves updated state. It upgrades apps in the existing namespace; it does not replace Kubernetes.

Admin CLI only (no app version change)

When release notes say only the admin CLI changed, run gt_ai_os_ensure_admin_cli_version via gt-ai-os-admin-env.sh (same pattern as step 1 above). The cluster app version stays the same until you run update --to ….

Rollback

sudo env GT_AI_OS_ADMIN_CONFIG_DIR=/var/lib/gt-ai-os/admin \
  KUBECONFIG=/etc/rancher/rke2/rke2.yaml \
  PATH="/var/lib/rancher/rke2/bin:/usr/local/bin:$PATH" \
  gt-ai-os-admin rollback --yes --namespace gt-ai-os-prod --to <older-tag>

sudo env GT_AI_OS_ADMIN_CONFIG_DIR=/var/lib/gt-ai-os/admin \
  KUBECONFIG=/etc/rancher/rke2/rke2.yaml \
  PATH="/var/lib/rancher/rke2/bin:/usr/local/bin:$PATH" \
  gt-ai-os-admin validate --namespace gt-ai-os-prod

Troubleshooting

Symptom	Fix
the database system is shutting down during update	Cosmetic while CNPG restarts. Wait a few minutes; sign in when pods are ready.
text file busy installing gt-ai-os-admin	Use gt_ai_os_ensure_admin_cli_version via gt-ai-os-admin-env.sh; run with sudo env ….
401/403 downloading release assets	Confirm outbound HTTPS to github.com and ghcr.io.
no state stored for namespace	Wrong namespace or missing /var/lib/gt-ai-os/admin state — confirm install completed.
ImagePullBackOff after upgrade	Confirm ghcr.io/gt-edge-ai images for the target tag are reachable.
In-app update dashboard	See Updates for the Control Panel workspace (placeholder in interim builds).

After upgrade

1. Confirm URLs with gt-ai-os-admin report.
2. Sign in to Control Panel and tenant app; continue day-2 setup in Getting Started and QuickStart.
3. For release-channel context and future in-app orchestration, see Updates.

Related pages

• Updates
• Advanced Management
• Backup & Restore