Skip to content

26.10.1

Latest
Latest

Choose a tag to compare

View all tags
@mickours mickours released this 29 Oct 16:07
· 10 commits to master since this release
26.10.1
34955f9

We are proud to announce the release of:

✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨

Ryax 26.10.1

✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨ ✨

A security patch release on top of 26.10.0. It fixes the advisories published since 26.10.0 and changes no behaviour or API, so upgrading needs no extra steps.

Security fixes

  • brotli 1.2.0 → 1.2.1 (CVE-2026-111497, out-of-bounds write, CVSS 9.8) in the worker-k8s, worker-ssh-slurm and action-builder images, by moving nixpkgs nixos-26.05 to 08d3a97. That also brings glibc 2.42-103.
  • gdb is gone from the worker images. Nothing used it, and in worker-k8s it was the only reason the image shipped curl and brotli. To debug a worker pod, use kubectl debug with an ephemeral container.
  • dnspython 2.8.0 → 2.9.0 in the Runner, the workers and the action builder (GHSA-45c3-73f7-pv4m).
  • msgpack 1.2.1 → 1.2.4 in the default actions (GHSA-j586-36cw-2gc2).

Documentation

  • The node pool API examples in the README and the GPU node pools guide now send the architecture field that 26.10.0 requires.

The 26.10.0 release notes follow, including its upgrade steps if you are coming from 26.9.0 or earlier.

Ryax 26.10.0

GPU requests by model, memory and share of a card, a random initial admin password, a broker and a filestore that no longer depend on Bitnami, and a round of fixes for imports, stuck executions and failed deployments.

New features

GPU scheduling by model, memory and share of a card

Ryax used to know a GPU only as a count, plus an opaque MIG profile name that had to match exactly. This release describes GPUs the way users think about them, from the action that asks to the node pool that answers.

  • Actions can ask for a kind of GPU. In ryax_metadata.yaml, spec.resources.gpu still takes a number, and now also takes an object with a count, a brand, a model, a memory (with a unit, for example 40G) and a compute_fraction (a share of one card, between 0 and 1, exclusive of 0). Everything in it is optional and describes one GPU: count: 2 with memory: 40G means two cards of 40 GB each, not 80 GB across two. gpu: 2 keeps its meaning of two GPUs of any kind. Brand and model must match the catalog; an action that names one no node pool has simply finds no place to run. A request with an unknown key, or a memory without a unit, is rejected when the action is scanned.
  • Node pools declare their card. A GPU node pool now records its catalog model next to its partition (full or a MIG profile), so Ryax knows how much memory and compute one GPU of the pool really gives. The new GET /api/runner/gpu-models?search=a100 1g lists the catalog as one flat, searchable list (brand, model, partition, memory, compute share), and the node pool form in the web interface now picks the GPU with one searchable select. Existing node pools keep working: one with no recorded model stays eligible for every request, and is simply treated as of unknown size, never as too small. Fill in the model to get precise matching.
  • Placement takes any partition that is big enough, and the tightest one by default. A recommendation of "3g.40gb" used to demand exactly that profile and fail when no pool had it; now any partition that meets the request will do. New setting RYAX_SCHEDULER_GPU_FIT_POLICY on the Runner: best_fit (the default) picks the pool that wastes the least of a card, leaving large partitions free for the actions that need them; first_fit keeps the previous behaviour of taking the best-scoring pool that fits. GPU requests are also now checked against the pool's CPU, memory and time, which they never were.
  • IntelliScale recommends a share of a card, not a MIG profile. It now recommends GPU memory (the observed peak plus 10%) and a compute share, and Ryax rounds them up to a partition your pools actually offer. This also works on cards that do not split in sevenths such as the A30.
  • Recommendations are learned per piece of hardware, not per site. A recommendation is only valid for the machine it was measured on, so IntelliScale now keeps one model per GPU model (for the GPU share) and per instance type (for CPU and memory). Two node pools of one site with different cards no longer contaminate each other, and two sites with the same card now learn together. The Runner asks for the recommendation after it has picked a candidate pool, one answer per pool, instead of merging every site's answer beforehand. Nothing fragments on upgrade: pools with no recorded GPU model, and HPC sites, share one model as before.
  • Autoscaled GPU nodes can be kept out of use until they are ready. An autoscaled GPU node is reported Ready long before its driver and MIG layout are in place, so actions landing on it got a whole GPU instead of their slice. The Kubernetes worker chart can now hold new GPU nodes behind a startup taint and release them only when the GPU stack is usable and the MIG layout is the one the pool asked for. Off by default: see gpuReadiness and the new GPU node pools and MIG guide.
    (roadmap#1421,
    roadmap#1458,
    roadmap#1469,
    roadmap#1428)

Security and operations

  • The initial admin password is random. Every installation used to boot with the same user1 / pass1 — hard-coded in the service, never set by the chart, and published in the README and the install guide. The chart now generates one per installation into the ryax-admin-credentials secret, and helm install prints the command to read it back. The account is admin. Existing installations keep their users and passwords: the secret is only ever read when the user table is empty, which is the very first start. The Edit password window now warns that changing the password does not update that secret.
    (roadmap#1423)
  • helm install prints its notes. NOTES.txt had always sat at the chart root rather than in templates/, where Helm is the only place it looks, so no install had ever printed anything. It now carries the admin credentials command and the Grafana one.
  • The broker no longer runs on a Bitnami image. It ran on bitnamilegacy/rabbitmq, which Bitnami no longer updates. It is now a RabbitmqCluster run by the official RabbitMQ Cluster Operator (2.23.0) on the official rabbitmq:4.3.6-management image. The chart deploys the operator in the Ryax namespace, watching that namespace only, and it needs neither cluster-wide rights nor cert-manager. The services reach the broker at the same address with the same credentials. This needs one command before upgrading, see below.
    (roadmap#1320)
  • The filestore no longer runs MinIO. MinIO's community edition is no longer
    maintained, and the chart ran it from the Bitnami bitnamilegacy/minio image. The
    filestore is now versitygw v1.8.0, a small S3
    gateway that keeps every object as a plain file on its volume. On a test cluster it
    answers Ryax's requests faster than MinIO did (about 3x on small writes, 1.3x on small
    reads) with a tenth of its memory. The services reach it at the same address
    (ryax-minio:9000) with the same credentials (ryax-minio-secret), and the upgrade
    copies the existing objects into it.
    (roadmap#1471)
  • The Authorization: Bearer <token> header works on every service. The Authorization and Repository services answered 401 to the form our own README documents, while Runner and Studio accepted it. All four now agree, and a header that is only Bearer answers 401 on Studio instead of 500.
    (roadmap#1453)
  • Deploying a workflow that is already deploying returns 409 with its state. POST /workflows/{id}/deploy answered 400 for this, the same code as for an invalid workflow, so clients had to match the error text. It now answers 409 {"error": "Workflow can't be deployed at this status", "deployment_status": "Deploying"}. An invalid workflow still gets 400. See the upgrade notes.
    (roadmap#1467)

Bug fixes and Improvements

  • An install reached through a proxy or by IP answers again. 26.9.0 restricted the Ingresses to global.tls.hostname, so any other name got a 404. They are now restricted only to global.ingress.hosts, which is empty (any host) by default. The documentation gained a how-to for running Ryax behind a proxy.
    (roadmap#1448)
  • Importing a workflow keeps its addons. An imported workflow was valid and showed Deployed, but its HTTP services had no Ingress and every URL answered 404, because addons were dropped. Export now writes the editable addon values (addons_inputs_values), and import restores them; packages exported before this release still import and get the addons' defaults. A value for an addon parameter the action does not have now answers 400 instead of being silently dropped.
    (roadmap#1464)
  • Importing a workflow with a numeric or directory input value no longer fails with a 500. A value such as max_tokens: 1024 or a directory input in a re-imported export now imports correctly, and a bad package answers 400. The deprecated table input/output type is removed; existing values are converted to file.
    (roadmap#1465,
    roadmap#1468)
  • An HTTP service trigger is no longer killed at startup. Addon inputs and resources were frozen at the first deploy of an action and shared by every workflow using it, so a workflow using the HTTP addon on an action first deployed without it lost the addon's inputs. Each workflow action now owns its own, and two workflows can use one action with different addons and resources. Addon values are merged parameter by parameter: a value set in the interface beats the action's, which beats the addon default.
    (roadmap#1463)
  • A workflow can be undeployed when it has several running deployments. Undeploy used to fail and leave Studio in "Undeploying", with the portal and trigger views answering 500. It now stops all of them, and a stopped deployment is never brought back to running by a late trigger event. The leftover PAUSED state, which nothing has set since 26.7.0, is retired.
    (roadmap#1337)
  • A "Run once" no longer blocks the next deploy of its workflow. When the trigger finished within seconds, its stop could be overwritten by the late "started" event. The deployment then stayed running forever, and every later deploy of that workflow was silently ignored, leaving Studio on "Deploying". Start, started and stop are now serialised per deployment, and a stopped deployment can no longer be set back to running. The upgrade repairs deployments already stuck this way.
    (roadmap#1485)
  • The workflow page follows a "Run once" to its end. It stayed on "Stop" after the run had finished, because it stopped refreshing the deployment's status. It now keeps refreshing it while the workflow is deployed.
    (roadmap#1491)
  • The Kubernetes worker survives a message backlog. After a broker outage it took every queued message at once, one database connection each, which OOM-killed its PostgreSQL. It now handles at most 10 messages at a time (RYAX_MESSAGING_PREFETCH_COUNT, 0 for no limit). Its metrics collection is also back to one fetch at a time, instead of piling up requests when the cluster is slow.
    (roadmap#1477,
    roadmap#1490)
  • The node pool form says why a value is refused. CPU, memory, scores and maximum time now show their limits, and the node pool API returns the architecture in the same lowercase form the form uses.
    (roadmap#1483)
  • A failed deployment says why. A deployment whose trigger failed (no site registered, trigger error, trigger cancelled by itself, deployment that could not even be created) stayed in "Deploying" forever or went back to "not deployed" with no explanation. It now fails and Studio shows The trigger '<name>' failed: <reason>.
    (roadmap#1445,
    roadmap#1466)
  • Executions that outlive their time allotment are cancelled. An execution whose Worker crashed or never reported its end stayed running forever, holding its resources. The Runner now cancels any action execution still running 5 minutes past its allotment (RYAX_EXECUTION_REAPER_GRACE_SECONDS, checked every 60 seconds, RYAX_EXECUTION_REAPER_INTERVAL_SECONDS; a grace of 0 disables it). Triggers, which have no allotment, are not touched.
    (roadmap#1299)
  • Stopping a workflow ends all its executions. Trigger runs of an http trigger stayed RUNNING when the trigger ended gracefully or crashed, and executions stayed STOPPING when the Worker did not know them or had restarted. They now end as cancelled, and no longer enter the retry chain. Executions already stuck are not repaired and need a one-off cleanup.
    (roadmap#1473)
  • Execution logs come out in the order the action wrote them. The Workers now number each batch of log lines and the Runner reads them in that order. Logs stored before the upgrade keep their previous order.
    (roadmap#1449)
  • Upgrades no longer deadlock on Grafana and MinIO. Their ReadWriteOnce volume stopped the new pod from starting while the old one still held it, so helm upgrade --wait timed out and marked the release failed. Both now replace their pod instead of rolling it.
    (roadmap#1451)
  • The registry garbage collector works again. It pointed at a configuration path that does not exist since the registry moved to v3, and hid the failure behind a success message, so no image was ever collected. A failing run now fails its Job.
    (roadmap#1294)
  • Security and dependencies. The web interface's advisories are cleared (Angular 21.2.25, axios 1.20, undici 6.29 and 7.30, piscina 5.3, adm-zip 0.6.1, brace-expansion 2.1.6; the unpatched braces advisory only affects the build tooling and is tracked), and Python dependencies and nixpkgs (26.05) are refreshed across all services. The container images move to glibc 2.42-100 and OpenSSL 3.6.5, clearing their high-severity CVEs. Observability charts: kube-prometheus-stack 89.x (Grafana 13), Loki 7.5, Alloy 1.14, Traefik 42.1.
  • Documentation. New GPU node pools guide, rewritten IntelliScale reference, global.ingress.hosts, proxy configuration and the RabbitMQ operator in the install, airgap and ArgoCD guides.

Upgrade to this version

Restore your values file if you do not have it:

helm get values -n ryaxns ryax --output yaml > values.yaml

Admins should take care of the following elements when upgrading to this version:

  • ⚠️ If you set global.tls.hostname, Ryax answers for every host again. 26.9.0
    restricted the Ingresses to that name; they are now restricted only to the names in
    global.ingress.hosts, which is empty (any host) by default. To keep Ryax to its own
    names, typically on a cluster shared with other applications, list them:

    global:
  ingress:
    hosts: ["ryax.example.com"]

    If you keep global.tls.hostname as well, it must be one of these hosts, or the chart
    refuses to render. Installs that set neither value need nothing. More in
    Install Ryax with ArgoCD › Routing.

  • A new GitOps install needs one more secret. With global.secrets.create=false,
    create ryax-admin-credentials (keys admin-user and admin-password) before the
    first sync, or set authorization.adminUsername and authorization.adminPassword.
    Without it the authorization pod stops with No initial admin password configured.
    An existing installation needs nothing — it never seeds again, and both
    secretKeyRefs are optional. The service also no longer has a built-in fallback for the initial user name (now admin when unset, not user1) nor for the JWT signing key: the chart has always injected the latter from api-jwt-secret-key, but a deployment that starts the services without the chart must now set RYAX_JWT_SECRET_KEY.

  • ⚠️ Apply the RabbitMQ CRD before upgrading. The broker moves to the RabbitMQ
    Cluster Operator, whose RabbitmqCluster CRD ships in the chart, and Helm never
    installs a CRD on an upgrade. Without it the upgrade stops before changing anything,
    and prints this command:

    kubectl apply --server-side -f https://gitlab.com/ryax-tech/ryax/ryax-engine/-/raw/26.10.0/charts/ryax/subcharts/rabbitmq/crds/rabbitmqclusters.rabbitmq.com.yaml

    Offline, take it from the chart package instead:

    tar -xzOf ryax-engine-26.10.0.tgz ryax-engine/charts/rabbitmq/crds/rabbitmqclusters.rabbitmq.com.yaml \
  | kubectl apply --server-side -f -

    --server-side is required: the CRD is too large for a client-side apply. ArgoCD
    applies the CRD itself, with the ServerSideApply=true the reference Application
    already sets.

  • The broker is replaced, not upgraded, and starts empty. Helm removes the Bitnami
    StatefulSet and the operator starts ryax-broker-server-0 in its place, about a minute
    later. The services keep the same address (ryax-broker:5672) and the same
    credentials (ryax-broker-secret), and reconnect by themselves. There is nothing to
    migrate: Ryax publishes its messages as transient, so a broker restart has always
    dropped whatever was still queued. As for any upgrade, run it while no workflow is
    running. The services retry on their own during the switch, which took about a minute
    on a test cluster. Once Ryax is back, delete the old broker volume, if there is one
    (there is none with rabbitmq.persistence.enabled: false, as in minimal.yaml):

    kubectl -n ryaxns delete pvc data-ryax-broker-0

  • The rabbitmq: values now configure the new broker. persistence.*,
    resources, tolerations, nodeSelector, affinity, priorityClassName and
    metrics.enabled keep their meaning. Every other Bitnami key (auth.*, image.*,
    clustering, plugins, ...) is ignored: remove them. A rabbitmq.image.repository
    still naming a Bitnami image stops the render. The broker password is always the one
    in ryax-broker-secret, so a rabbitmq.auth.password set by the old troubleshooting
    guide does nothing. The broker also follows global.tolerations, nodeSelector and
    affinity now.

  • Uninstalling now takes one more step. helm uninstall removes the operator at
    the same time as the broker, so nothing clears the RabbitmqCluster finalizer: the
    broker pod keeps running, and helm uninstall --wait times out. Delete the broker
    first, while the operator still runs:

    kubectl -n ryaxns delete rabbitmqcluster ryax-broker
helm uninstall ryax -n ryaxns

    If an uninstall is already stuck, clear the finalizer:
    kubectl -n ryaxns patch rabbitmqcluster ryax-broker --type merge -p '{"metadata":{"finalizers":[]}}'.

  • If the cluster already runs a RabbitMQ Cluster Operator that watches the Ryax
    namespace, set rabbitmq.operator.enabled: false. Two operators would reconcile the
    same broker.

  • Multi-site with Skupper: the ryax-broker-ext and ryax-minio-ext connectors
    copied the pod selectors of the old broker and of MinIO when they were created. After
    the upgrade the first matches no pod, and the second matches the old MinIO, which
    refuses their connections. Recreate both on the main site right after the upgrade,
    before running workflows on remote sites:

    skupper -n ryaxns connector delete ryax-broker-ext
skupper -n ryaxns connector create ryax-broker-ext 5672 --workload service/ryax-broker
skupper -n ryaxns connector delete ryax-minio-ext
skupper -n ryaxns connector create ryax-minio-ext 9000 --workload service/ryax-minio

  • GitOps (global.secrets.create=false): ryax-broker-cookie is no longer read,
    delete it whenever you like. ryax-broker-secret keeps its keys; its broker-user
    and rabbitmq-password now seed the broker's user, so they must match the broker
    URL as before. With automated pruning off, prune the old broker resources in the same
    sync: the operator cannot create its ryax-broker Service while the Bitnami one is
    still there.

  • ⚠️ The filestore moves from MinIO to versitygw, and the upgrade copies its
    objects.     Helm keeps the old MinIO, ryax-minio, running on its volume, and marks
    that volume so that neither Helm nor ArgoCD ever deletes it. The new filestore,
    ryax-filestore, gets a volume of its own, the size of MinIO's. Before it starts
    serving, its pod copies every object out of MinIO, checks that both sides list the same
    objects and sizes, and leaves a marker so that it never copies again. The address and
    the credentials do not change.

    Downtime: the services cannot reach the filestore until the copy is over. Without
    a pre-copy (below), count the time to read the whole MinIO volume once: on a test
    cluster, 750 MiB in 3,000 objects took a few seconds, and a large volume on network
    storage takes minutes. On a local k3s upgrade of a 26.9.0 install holding 58 MiB in
    410 objects, the copy itself took a second, the filestore was unreachable for 15 to
    50 seconds while MinIO restarted, and the whole of Ryax was back within 3 minutes,
    the broker switch being the longest part. The runner and studio restart while they
    wait, and their restart back-off can add up to five minutes once the copy is over.
    They reconnect on their own; to skip the back-off, restart them once the filestore is
    Ready:

    kubectl -n ryaxns rollout status deploy/ryax-filestore --timeout=24h
kubectl -n ryaxns rollout restart deploy/ryax-runner deploy/ryax-studio

    Follow the copy with
    kubectl -n ryaxns logs -f deploy/ryax-filestore -c migrate-from-minio. The new
    volume's storage class must support user extended attributes, as ext4 and xfs do.

    If MinIO ran without persistence (minio.persistence.enabled: false), it has no
    volume to copy from, and its objects go away with it, as they did whenever its pod
    restarted. The minio: values are ignored now: remove them, but carry a
    minio.persistence.storageClass over as
    filestore.migration.legacy.persistence.storageClass, as the class of MinIO's volume
    cannot change. To give the filestore more room than MinIO had, set
    filestore.persistence.size.

  • Optional: pre-copy the objects to shorten the upgrade. While 26.9.0 still runs,
    this copies the objects into the volume the filestore will use, and the upgrade then
    only copies what changed since. It can run several times, and it can be interrupted:
    the copy at the upgrade makes the result exact whatever it left. Use the same values
    file as for the upgrade:

    SIZE=$(kubectl -n ryaxns get pvc ryax-minio -o jsonpath='{.spec.resources.requests.storage}')
helm template ryax oci://registry.ryax.org/release-charts/ryax-engine --version 26.10.0 \
  -n ryaxns -f values.yaml \
  --set filestore.migration.precopy.enabled=true \
  --set filestore.persistence.size="$SIZE" \
  --show-only charts/filestore/templates/precopy.yaml > precopy.yaml
kubectl -n ryaxns delete job ryax-filestore-precopy --ignore-not-found
kubectl -n ryaxns apply -f precopy.yaml
kubectl -n ryaxns wait --for=condition=complete job/ryax-filestore-precopy --timeout=24h
kubectl -n ryaxns logs job/ryax-filestore-precopy | tail -n 1
kubectl -n ryaxns delete job ryax-filestore-precopy

    It creates the ryax-filestore volume, which the upgrade then takes over, and a Job
    that reads MinIO while Ryax keeps working. The upgrade refuses to start while that Job
    runs. Delete the Job once it is done, as above: as long as it exists, the volume
    cannot be deleted, which a rollback or an uninstall does. The pre-copy only shortens
    the upgrade when the volume is large: on the test cluster above, both copies took
    about a second. Offline, render it from the chart package (helm template ryax ryax-engine-26.10.0.tgz ...): it only uses the MinIO image 26.9.0 already ran.

  • Once 26.10.0 runs, decommission the old MinIO. It only serves the copy, and
    makes a rollback possible until then. First check that the copy is done:

    kubectl -n ryaxns exec deploy/ryax-filestore -c versitygw -- cat /data/.ryax-migrated-from-minio

    It prints the date of the copy. Then remove MinIO, and delete its volume:

    helm upgrade ryax oci://registry.ryax.org/release-charts/ryax-engine:26.10.0 \
  -n ryaxns --reuse-values --set filestore.migration.enabled=false
kubectl -n ryaxns delete pvc ryax-minio

    The chart refuses to remove MinIO while no filestore pod has finished the copy
    (filestore.migration.allowDecommissionWithoutCopy=true forces it), and refuses
    filestore.migration.enabled=false on an upgrade from 26.9.0, which would delete
    MinIO's volume before the copy. Like any upgrade, the decommission restarts the
    runner and studio, and it restarts the filestore, unreachable for a few seconds.
    After the decommission, a rollback to 26.9.0 is no longer possible without
    restoring MinIO's volume from a backup.

  • Rolling back to 26.9.0 before the decommission brings MinIO back on its volume,
    untouched, and deletes the filestore and its volume. Objects written since the
    upgrade are lost, unless you copy them back into MinIO first, while 26.10.0 still runs
    and no workflow runs. That copy rewrites every object, not only the new ones, so it
    takes about as long as the copy at the upgrade. Then delete the broker while its
    operator still runs, for the Bitnami broker cannot take its ryax-broker Service
    back otherwise, and roll back:

    kubectl -n ryaxns exec deploy/ryax-minio -- sh -c 'export HOME=/tmp
  mc=/opt/bitnami/minio-client/bin/mc
  $mc alias set new http://ryax-minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
  $mc alias set old http://localhost:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
  $mc mirror --overwrite new/ryax-filestore old/ryax-filestore'
kubectl -n ryaxns delete rabbitmqcluster ryax-broker
helm rollback ryax <the 26.9.0 revision> -n ryaxns

    Without the delete rabbitmqcluster, the rollback stops part-way on
    no Service with the name "ryax-broker" found. Do not run a failed rollback again:
    each attempt leaves more of both versions behind. Upgrade to 26.10.0 again with the
    same values file, wait for every pod to be Ready, and roll back as above. Upgrading
    again later copies MinIO afresh. A ryax-worker-k8s already on 26.10.0 can stay
    there: on the test cluster it ran workflows with the rolled-back engine.

  • GitOps (ArgoCD, Flux): the chart cannot see the cluster, so it always renders
    the old MinIO and its volume while filestore.migration.enabled is on, the default.
    The upgrade copies as above. If you changed minio.persistence.size or
    storageClass, set them as filestore.migration.legacy.persistence.size and
    storageClass, or the sync fails on the volume, which cannot shrink. To decommission,
    check the marker as above, then set filestore.migration.enabled: false: ArgoCD
    removes MinIO but keeps its volume (Prune=false), which you delete by hand. A new
    GitOps install     has nothing to migrate: set filestore.migration.enabled: false from
    the start.

  • global.security.allowInsecureImages is gone. It only served the Bitnami
    charts, and the engine chart no longer has any. Remove it from your values; it is
    ignored.

  • Grafana 13 (kube-prometheus-stack 89). The bundled Grafana runs the -distroless image with a read-only root filesystem and an emptyDir on /tmp. GF_*__FILE environment variables and GF_INSTALL_PLUGINS are no longer supported. The Ryax values set none of them, and the plugins (grafana-piechart-panel, grafana-clock-panel, vonage-status-panel) are now preinstalled synchronously at startup, so no values change is needed, but check that the Grafana pod starts. If your own overrides set GF_*__FILE, GF_INSTALL_PLUGINS or an extraVolumeMounts on /tmp, migrate them first. The rollout also replaces the Grafana and MinIO pods instead of rolling them, so both are briefly unavailable during the upgrade.

  • Kubernetes worker: the MIG auto-labeler is removed. The values config.MIG and labeler are gone, along with the node-labeler DaemonSet. If you relied on it to turn a gpu-pool-mig-* node label into nvidia.com/mig.config, label the GPU nodes with nvidia.com/mig.config yourself (cloud node-pool labels are the usual way), as the NVIDIA GPU Operator's MIG Manager expects. Remove both keys from your worker values. To keep actions off GPU nodes that are not ready yet, see the new opt-in gpuReadiness values: enable it before adding the startup taint to the node pool, as a tainted pool with the gate off never runs anything.

  • ⚠️ GPU node pools created before this version: recreate them. Existing GPU pools, MIG ones included, have no GPU model: they keep running, but Ryax cannot match them on model, memory or share of a card, and the web interface shows their GPU as -. As a pool's GPU cannot be changed after creation, replace each one:

    1. Recreate it in Infrastructure with the same settings, picking its card in the GPU select.
    2. Add the new pool's id to the Kubernetes worker values next to the old one, with the same selector, and upgrade the worker.
    3. Pin the actions pinned to the old pool to the new one instead (Studio, Constraints › Sites and Node Pools), and redeploy their workflows. Also redeploy the workflows whose trigger runs on the old pool.
    4. Archive the old pool.
    5. Once nothing runs on it any more, remove the old entry from the worker values and upgrade the worker.

    ⚠️ Archiving a pool does not touch deployed workflows: an action still pinned to it keeps running on it, and a trigger running on it stays there. Redeploying a workflow without changing the pin drops it, and the action can then run on any pool. A worker asked to run on a pool it no longer lists runs the execution on the first pool of its values. Hence the order above. (roadmap#1492 will make archiving handle this.)

  • GPU placement defaults. Placement now defaults to best_fit. To keep the previous behaviour, set RYAX_SCHEDULER_GPU_FIT_POLICY=first_fit through runner.extraEnv.

  • Kubernetes 1.30 or later is required. The bundled Traefik chart moved to 42.x, which refuses older clusters. Check kubectl version before upgrading.

  • API users:

    • POST /workflows/{id}/deploy returns 409 (with deployment_status in the body) instead of 400 when the workflow is already deploying, deployed or undeploying. A client that matched the 400 text must handle 409 instead.
    • Node pools take and return gpu_count, not gpu (the stored value is migrated). A client that still sends gpu when creating a node pool gets a 422.
    • Creating a node pool now requires architecture (x86_64 or arm64). A client that leaves it out gets a 422; the web UI already sends it.
    • Node pools return architecture in lowercase (x86_64, arm64), as the create request takes it. It was X86_64 before.
    • GET /api/runner/node-pools/{id} answers 404 for an archived node pool.
    • New root endpoints /api/runner/node-pools (filters on GPU, site, CPU and memory; the GPU is named by gpu_config_id) and GET /api/runner/gpu-models returns a flat list. The nested /sites/{id}/node-pools routes remain, marked deprecated.
    • GET /modules/{id} and workflow action views now return gpu_brand, gpu_model, gpu_memory_gb and gpu_compute_fraction in resources.
    • The table input/output type no longer exists.
    • Authorization: Bearer <token> is accepted by all services.

  • Registry garbage collection now actually runs. The first scheduled run after the upgrade removes every untagged image left in the registry, and a failing run now shows as a failed Job.

Then run the upgrade:

helm upgrade ryax oci://registry.ryax.org/release-charts/ryax-engine:26.10.0 \
  -n ryaxns \
  -f values.yaml

And each worker with its own values:

helm upgrade ryax-worker-k8s oci://registry.ryax.org/release-charts/ryax-worker-k8s:26.10.0 \
  -n ryaxns \
  -f worker.yaml
Assets 2
Loading