26.8.0rc1

Breaking Changes

• The TUI installer no longer hardcodes the wildcard app/storage base domains; installs that pass --fqdn-prefix must now also pass --app-base-domain and --storage-base-domain, otherwise those hosts fall back to --public-facing-address. (#12206)
• The GraphQL and REST v2 BinarySizeInfo type now returns the exact byte count as a string field expr (replacing the integer value), lifting the 2 GiB GraphQL Int limit that made larger sizes fail to serialize; clients must read expr as a string on resource policy maxQuotaScopeSize, vfolder usedBytes/maxSize, and resource preset/allocation sharedMemory. (#12638)
• Schedule all workloads within a single tick in priority order instead of dropping lower-priority workloads (#12668)
• The agent no longer reads the sgroup/<name> etcd config scope. Deployments that set per-scaling-group etcd overrides must move those values to the GLOBAL or NODE scope. (#12906)
• The agent config key scaling-group is replaced by a nullable initial-resource-group-name that only seeds the resource group at the agent's first registration; agents with the removed key in agent.toml now fail at startup, and when unset the manager assigns the default resource group. (#13005)
• Drop the id-addressed app config fragment write endpoints (create, update, bulk-update) and the paginated scoped search from REST v2; use the scoped upsert and by-names endpoints instead. (#13196)

Features

Reconciler-Based Idle Checking (BEP-1054)

Built out the reconciler-based idle checking stack defined in BEP-1054: DB-backed per-checker deadlines, the assignment-sync / grace-period / judgment / expiry-sweep stages wired into the Sokovan coordinator, and the full management surface (repository, service, GraphQL, REST v2, SDK, CLI) for idle checker definitions and their scope assignments.

• Wire IdleCheckerRepository into the idle-check reconciler source to fetch a normalized session/scope/checker snapshot (#12398)
• Judge idle sessions in the reconciler idle-check handler by driving each checker type's batched judgment, merging per-session idle reports with each checker's reason (#12637)
• Add a reconciler-based session lifetime checker. (#12821)
• Add a reconciler-based network timeout idle checker using last-access and active-connection signals. (#12878)
• Add the session_idle_checks table storing per-checker idle deadlines for the reconciler-based idle checker (BEP-1054) (#12918)
• Add the expiry-sweep source that fetches expired session idle-check deadlines for the reconciler-based idle checker (BEP-1054) (#12925)
• Add idle-check expiry sweep termination handling. (#12975)
• Register the idle-check expiry sweep stage with the reconciler. (#12988)
• Require every session idle check to have an expiration deadline. (#13023)
• Add idle-check assignment synchronization source. (#13038)
• Add idle-check lifecycle phases. (#13040)
• Add the idle-check assignment-sync Handler and Applier for session row lifecycle updates. (#13045)
• Add initial grace period source for idle checker sessions. (#13068)
• Add the initial grace period handler and applier for reconciler-based idle checks. (#13090)
• Wire all idle-check reconciliation stages into the Sokovan coordinator. (#13100)
• Implement the Sokovan idle-check applier to persist checker judgments. (#13108)
• Support utilization-based idle checks with expiration-aware judgments. (#13133)
• Add super-admin GraphQL v2 operations for managing global idle checker definitions with session-lifetime settings. (#13134)
• Add repository CRUD support for idle checker definitions. (#13135)
• Add GraphQL v2 operations for managing idle checker bindings that associate global idle checkers with domain, project, or resource group scopes. (#13192)
• Add admin API and CLI management for idle checker configurations. (#13208)
• Add the idle checker assignment v2 SDK client and bai idle-checker-assignment CLI commands. (#13209)
• Add the idle checker assignment repository layer with write-path scope validation and RBAC entity registration. (#13212)
• Expose idle checker assignment management through the service layer, GraphQL API, and REST v2 endpoints. (#13214)
• Support network timeout and utilization specifications in idle checker CRUD APIs. (#13230)

Scoped App Config (BEP-1052)

Completed the scoped app-config redesign: the merge rank moved to the admin-owned allow-list, fragments became RBAC-scoped, and the read / write / search / upsert surfaces landed across the repository, service, REST v2, and GraphQL layers.

• Add the app_config read service: resolve a user's merged AppConfig for one or many config names by rank-ordering the visible public / domain / user fragments and deep-merging them. (#12359)
• Add bulk create / update / purge for app_config_fragment at the service layer (partial-success batches: each fragment write is authorized by its FK to the allow-list, and rejected or failed items are reported per-item rather than failing the whole batch). (#12401)
• Add app_config_fragment bulk repository operations (bulk_create/bulk_update/bulk_purge with per-item partial success): bulk create relies on the allow-list FK so an item with no allow-list row is rejected per item, bulk update/purge report missing targets per item, and the AppConfigFragmentBulkWriteResult data type carries the partial result (#12426)
• Move the app config merge rank from app_config_fragments to the admin-owned app_config_allow_list with per-scope-type defaults (public=100, domain=200, user=300) (#12516)
• Expose the app config allow-list merge rank on the v2 REST/GraphQL APIs and CLI — optional create input (scope-type default when omitted), node field, order field, and a new admin update operation (PATCH /v2/app-config-allow-list/{id}, adminUpdateAppConfigAllowList, ./bai admin app-config-allow-list update --rank) for re-ordering the merge (#12517)
• Cascade app config deletion down the definition → allow-list → fragment subtree via ON DELETE CASCADE FKs, so a fragment can no longer exist without its allow-list entry (#12518)
• Add the AppConfigFragment visible-fragments repository query: fetch the public / domain / user fragments visible to a (domain, user) principal for one or many config names, rank-ordered via the allow-list, in a single query. (#12706)
• Bind user/domain AppConfig fragments to their RBAC scope on create and unbind them on purge, so a fragment's ownership is resolvable when authorizing writes (public stays global-scoped, with no association) (BEP-1052). (#12826)
• Grant APP_CONFIG_FRAGMENT permissions to the roles seeded from the RBAC role fixture, so app config fragment writes are authorized on a freshly installed environment. (#12837)
• Register APP_CONFIG_FRAGMENT as an RBAC resource type and backfill its write permissions onto existing write-capable roles, so app config fragment writes can be authorized by RBAC (BEP-1052). (#12856)
• Add the AppConfig fragment write REST v2 API: create / get / update / purge plus bulk update and bulk delete of raw config fragments, open to any authenticated user and gated by RBAC per scope (BEP-1052). (#12928)
• Add the AppConfig fragment search REST v2 API: a superadmin system-wide paginated search and an RBAC-authorized search of one scope's fragments (BEP-1052). (#13020)
• Expose AppConfig fragments for reading over GraphQL: fragment get, admin search and scoped search queries, resolvable through the Relay node entry point (BEP-1052). (#13047)
• Add the scopedAppConfigFragmentsByNames and myAppConfigFragmentsByNames GraphQL queries to read the current app config fragment values at a scope by config name. (#13110)
• Add the upsertAppConfigFragments and myUpsertAppConfigFragments GraphQL mutations to upsert many app config fragments at a scope in one call. (#13115)
• Add scoped/upsert and scoped/by-names app config fragment endpoints to REST v2, with my/ variants acting at the caller's own user scope. (#13118)

Virtual Scope RBAC Ownership Model (BEP-1062)

Introduced the virtual scope ownership layer that replaces the recursive scope walk with a non-recursive scope -> virtual_scope -> entity resolution, including its tables, DB ops, permission-check APIs, and action validators.

• Add virtual scope ORM tables (virtual_scope, association_virtual, association_entity) for the RBAC virtual scope model. (#12503)
• Add virtual scope DB ops for the RBAC ownership layer: spec-based scope lifecycle that materializes each scope's virtual scope, plus idempotent scope-binding and entity-membership writes. (#12528)
• Add a repository function to verify a user's effective permission on an entity through the virtual-scope chain, clipping by the permission_cap on each hop. (#12814)
• Register a scope as an entity member of its own virtual scope on creation, so access to the scope object itself resolves through single-path RBAC permission resolution. (#12871)
• Let an RBAC entity creator carry no scope (scope_ref=None) so a GLOBAL entity outside the RBAC scope hierarchy is created with no scope association, and add bulk_create_scoped_partial / bulk_purge_scoped_partial for scoped bulk writes that need per-item partial success (BEP-1052). (#12914)
• Materialize a project's virtual scope on creation so it can own child entities through the virtual-scope chain. (#12931)
• Add ensure-virtual-scope RBAC ops that bind a scope to its own (and optionally its parent's) virtual scope, so ownership resolves through the single virtual-scope path. (#12934)
• Add virtual-scope-chain RBAC action validators (scope / single-entity / bulk) resolving permissions without the recursive scope-walk (#13078)
• Add per-target virtual-scope-chain permission check APIs (single-entity / bulk / scope) to the permission-controller repository, with a dedicated scope-target check key matching permission rows on the acted-on entity type (#13107)
• Add an upsert_scoped RBAC write-op primitive that upserts a row (INSERT ON CONFLICT UPDATE) and binds a newly inserted row to its RBAC scope. (#13113)

Super-Admin User Impersonation (BEP-1058)

Added the ability for a super admin to run a request under a target user's permission context, recording the triggering and effective identities separately across audit logs, the event dispatcher, the action reporter, and the read/export surfaces.

• Add a triggered_user() identity context and resolve the X-BackendAI-Act-As header in the REST auth middleware, letting a super admin run a request under a target user's permission context (BEP-1058) (#12660)
• Propagate the triggered_user() identity through the event dispatcher so background handlers restore both the effective and triggering user, keeping audit consistency for work triggered during impersonation (BEP-1058) (#12701)
• Record audit-log actors as separate triggered_by (trigger) and acted_as (effective) identities, diverging only during super-admin impersonation (BEP-1058) (#12704)
• Expose the audit-log acted_as (effective/acting) identity through the REST v2, GraphQL, and export read surfaces so consumers can attribute an impersonated action to the effective user (BEP-1058) (#12732)
• Support filtering audit logs by acted_as (the effective/acting user UUID) across the v2 GraphQL / REST DTOs, repository, adapter, and the ./bai audit-log search --acted-as CLI option (BEP-1058) (#12748)
• Record triggered_by (the caller) and acted_as (the effective/acting user) separately in action reporter messages, so the reporter and the audit-log monitor agree during super-admin impersonation (BEP-1058) (#12758)
• Support filtering the audit-log CSV export by acted_as (the effective/acting user UUID), matching the triggered_by export filter (BEP-1058) (#12761)

Resource Group Registration by ID (BEP-1059)

Made the manager DB authoritative for agent-to-resource-group registration and moved sessions, kernels, and agents onto id-addressable resource group references ahead of the resource group PK swap.

• Add domain_id and resource_group_id columns to session rows and populate them at session creation (#12452)
• Add resource_group_id input to session creation APIs, deprecating the name-based resource_group field (#12513)
• Add non-nullable resource_group_id column to kernels, backfilled from the scaling group name, making kernel data id-addressable ahead of the resource group PK swap (#12557)
• Add non-nullable resource_group_id column to agents, backfilled from the scaling group name and kept consistent by heartbeat ingest and the modify-agent mutation ahead of the resource group PK swap (#12624)
• Add an is_default column to scaling groups with a partial unique index so that at most one resource group can be the default at a time. (#12885)
• Add a service to change an agent's resource group that also handles the sessions still running on it: it fails unless forced (so an admin can drain first), and on force it terminates those sessions before applying the change. (#12917)
• Remove the manager-to-agent update_scaling_group RPC push (#12992)
• Add a v2 agent resource-group change API (REST v2, GraphQL, SDK v2, CLI) that moves an agent to another resource group with a conflicting-session cleanup policy (terminate) and a force flag, returning the conflicting and terminating session lists (#13093)

Session Preemption (BEP-1055)

Added scheduler preemption: a scope-local job_priority axis, resource-group preemption settings, the PREEMPTED lifecycle state, and reserved placements that start once the victims' resources are freed.

• Add the PREEMPTED session status as the single lifecycle state for a preemption victim, branching by preemption mode to TERMINATED or PENDING after kernel cleanup. (#12623)
• Add an enabled toggle and a preemption_min_runtime duration to resource group preemption configuration, exposed via GraphQL and REST v2. (#12627)
• Add per-session job_priority, a scope-local preemption priority that ranks a user's own sessions independently of the global scheduling priority. (#12953)
• Load preemption victim candidates into the scheduling fetch for preemption-enabled resource groups, grouped per owner with per-agent reclaimable allocations for the preemption planner. (#13123)
• Preempted sessions are now either terminated or put back in the queue, according to the resource group's preemption mode. (#13126)
• Preempt lower-priority sessions of the same owner to admit starved pending sessions: the scheduler reserves the placement up front (RESERVED status) and starts it once the victims' resources are freed. (#13146)

DB Record Retention Management (BEP-1063)

Added a general-purpose retention layer where a super admin sets a retention period per data category and a leader-cron sweep purges older records referentially safely.

• Add a retention_policies table and RetentionCategory catalog as the storage foundation for DB record retention management. (#12951)
• Add a DB record retention repository that purges accumulated log, reconcile-history, login, roles/invitation, and usage records past their configured age boundary. (#12952)
• Add a super-admin v2 API (REST v2 + GraphQL) for managing per-category DB record retention policies. (#12954)
• Add DB record retention cleanup for the sessions category with ordered kernel-then-session deletion. (#12955)
• Add retention policy client SDK v2 and admin CLI v2 (bai admin retention-policy). (#12956)
• Add a leader-cron sweep that deletes DB records past each retention policy's configured age boundary. (#12957)
• Add dedicated retention cleanup for deployments and usage_buckets, purging destroyed endpoints (with their revisions, terminal routings and replica groups, and expired tokens) and expired usage buckets. (#12958)

Scheduling History APIs

Exposed kernel and replica-group scheduling histories end to end — query conditions, repository search scopes, DTOs and actions, REST v2 endpoints, GraphQL queries, SDK, and CLI.

• Add kernel scheduling-history queries to REST v2, the SDK, and the CLI (bai scheduling-history kernel search / search-scoped). (#12866)
• Add the models-layer query conditions and orders for kernel scheduling history, covering filters on id, kernel, session, phase, status transition, result and timestamps. (#12867)
• Add the kernel scheduling-history repository search scope, which scopes a query by session, by kernel, or by both, and rejects an empty scope rather than degenerating into a system-wide search. (#12868)
• Add the v2 DTOs and service actions for kernel scheduling-history search, forming the schema shared by REST v2 and GraphQL. (#12869)
• Add the kernel scheduling-history adapter and its REST v2 endpoints: a super-admin-only system-wide search, and a kernel-scoped search whose access is checked against the RBAC scope. (#12870)
• Add adminKernelSchedulingHistories and scopedKernelSchedulingHistories GraphQL queries, and a KernelV2.schedulingHistories field. (#12989)
• Add the models-layer query conditions and orders for replica-group scheduling history, along with the ReplicaGroupHistoryID identifier type. (#13056)
• Add the replica-group scheduling-history repository, with an admin search and a search scoped by the owning deployment. (#13057)
• Add the v2 DTOs and service actions for replica-group scheduling history, scoped by the owning deployment and filterable down to a single replica group. (#13058)
• Add the replica-group scheduling-history REST v2 endpoints: POST /v2/scheduling-history/replica-groups/admin/search and /replica-groups/scoped/search. (#13059)
• Add replica-group scheduling-history commands to the SDK and CLI (bai scheduling-history replica-group search / search-scoped --deployment-id <ID>). (#13060)
• Expose replica-group scheduling history over GraphQL via adminReplicaGroupHistories, scopedReplicaGroupHistories, and ModelDeployment.replicaGroupHistories (#13182)

Scheduler Dry-Run (Compute Schedule)

Added a dry-run scheduling probe that reports, per kernel, whether a session would fit a resource group's nodes before any provisioning, with per-kernel remediation hints across GraphQL, REST v2, SDK, and CLI.

• Add scheduler dry-run schedule action and value types. (#12496)
• Add the compute-schedule GraphQL schema (DTOs, types, and the computeSchedule query) for probing resource-group admission without provisioning. (#12629)
• Add a compute-schedule query that reports, per kernel, whether a session could be placed on a resource group's nodes before provisioning. (#12806)
• Add the computeSchedule (dry-run) GraphQL query that probes, without provisioning, whether each kernel of a would-be session fits a resource group's nodes and returns per-kernel remediation hints. (#12841)
• Add POST /v2/sessions/compute-schedule REST endpoint to probe session schedulability against a resource group without provisioning (#13014)
• Add ./bai session compute-schedule CLI command and SDK v2 function for the scheduler dry-run endpoint (#13017)
• Support designated agent IDs and an agent selection policy in the scheduler dry-run (compute-schedule) API across REST v2, GraphQL, SDK v2, and CLI, matching the semantics of the real scheduling path (#13080)

SessionGroup Placement (BEP-1064)

Added SessionGroup, an agent-level placement policy that spreads or packs a set of related sessions, and applied it to every deployment replica group.

• Add SessionGroup, the placement policy that keeps a set of sessions apart (spread) or together (pack) per agent, and give every deployment replica group one. (#13180)
• Route sessions now inherit the SessionGroup of the replica group they belong to, so a deployment's replica placement policy applies to every replica from the moment it is created. (#13183)
• Clean up the placement groups left behind by terminated replica groups during the deployments retention sweep (#13184)
• Place the sessions of a SessionGroup by its policy: spread keeps them on distinct agents and pack gathers them onto the same ones, either as a preference or as a strict requirement. (#13185)

Agent Re-architecture (BEP-1057)

Laid the container/VM-common foundation for the agent re-architecture: the ComputeBackend interface, an in-memory dummy backend, and a CLI to drive a running agent's kernel RPC directly.

• Add an ag kernel CLI to drive a running agent's kernel RPC directly (create/inspect/destroy and helpers) for verifying agent refactor parity. (#12628)
• Add the ComputeBackend interface and instance value types as the container/VM-common foundation for the agent re-architecture. (#12639)
• Add an in-memory dummy compute backend implementing the ComputeBackend instance-lifecycle interface for tests and local development. (#12644)

Model Serving & Deployment

Extended deployment revisions and the model service configuration surface, and hardened revision creation against malformed definition files.

• Default the model service shell to /bin/bash in deployment revision inputs so string start commands run under a shell when shell is omitted (#12622)
• Add --shell and --no-shell options to the deployment revision add v2 CLI to control model service shell wrapping (#12797)
• Support some, every, and none nested replica filters for deployments. (#12805)
• Expose runtime variant model defaults through REST and GraphQL APIs. (#12970)
• Reject revision creation when deployment definition files are malformed. (#13096)

Scheduling Policy Controls

Added per-request and per-keypair controls over how sessions are admitted and prioritized.

• Accept an optional agent_selection_policy in the v2 session enqueue API to override the resource group default per request (#13082)
• Enforce the max_pending_session_resource_slots keypair resource-policy limit at session enqueue, rejecting session creation that would exceed the pending-queue slot caps (#13085)
• Add max_priority to the keypair resource policy to cap the scheduler priority a user may request when creating a session (#13125)

GraphQL DataLoader Extension

Added DataLoader-backed nested fields so related entities can be fetched in a single query instead of a follow-up round trip.

• Add a DataLoader-backed runtimeVariant: RuntimeVariant nested field to the RuntimeVariantPreset GraphQL type so the runtime variant name can be fetched in one query alongside a preset list, instead of firing a separate runtimeVariants query. (#12708)
• Add DataLoader-backed project, domain, and resourceGroup nested fields to the ProjectUsageBucket GraphQL type so related entities can be fetched in one query, mirroring the sibling ProjectFairShare type. (#12713)
• Add DataLoader-backed user, project, domain, and resourceGroup nested fields to the UserUsageBucket GraphQL type so related entities can be fetched in one query, mirroring the sibling UserFairShare type. (#12714)

Action Framework Restructuring

Introduced self-contained scope and bulk action runners with their own processors, validators, and monitor abstractions.

• Add an independent scope action runner (processor, validator, and monitor abstraction) bound to the self-contained scope base action. (#12909)
• Add an independent bulk action runner (processor, validator, and monitor abstraction) bound to the self-contained bulk base action. (#12913)

Other Features

• Add a resourceAllocation field to the v2 GraphQL SessionV2 and KernelV2 types that reports requested / used / allocated resource slots aggregated from the resource_allocations table, where allocated persists after the session or kernel is freed or terminated (for post-termination statistics and billing). (#12546)
• Add a mapped-scope filter to the role search API so roles can be queried by the scope (domain / project / user) they are registered in. (#12563)
• Support filtering a role's scopes by scope type and scope id in the Role.scopes GraphQL field. (#12670)
• Add the allow_theme_mode webserver config option to expose the WebUI allowThemeMode flag, which gates the theme (family) selector and primary color customization in user settings (#12745)
• Add declarative conflict checks to the purger spec pattern, validated in a single query before deletion (#13041)

Improvements

Manager Layer Dependency Cleanup

Continued flattening the manager's layer graph — enum and value types moved down into the data layer, dead model/query surface removed, and the dependency direction is now enforced at build time by Pants visibility rules.

• Move session/kernel/network status enums down into the manager data layer to cut cyclic data->models import edges. (#12505)
• Move auth enums (UserRole/UserStatus, login-session status/result) down into the manager data layer to cut cyclic data->models import edges. (#12511)
• Move agent status and rbac permission-definition enums down into the manager data layer to cut cyclic data->models import edges. (#12515)
• Remove unused session row and repository helpers to reduce legacy manager model/query surface area. (#12527)
• Remove unused scheduler repository queries (get_schedulable_scaling_groups, _get_scaling_group_network_info, _resolve_image_info) and dead KernelRow methods and module helpers (#12595)
• Move deployment strategy schemas and their value types from the manager models/data layers down to a new shared common/schema module to cut the models↔data import cycle. (#12630)
• Make the manager data layer a pure leaf by moving the remaining vfolder, scaling group, and image value types down from models into data. (#12643)
• Remove the manager models layer's outward imports on config and clients so it depends only downward. (#12673)
• Move the database engine creation and lifecycle utilities from the manager models layer into the repository layer, decoupling models from database configuration. (#12677)
• Enforce manager (data -> views -> models, with config/clients forbidden from these layers) and common (dto -> schema) layer dependency direction at build time with Pants visibility rules. (#12718)

Scheduler Read Path

Rebuilt the scheduler read path on normalized snapshot loading and decoupled it from ownership scope and storage RPCs, so a resource-only probe no longer pays for context it never uses.

• Separate vfolder-mount resolution from the session spec-context fetch so scheduling callers that only need kernel resources can skip the storage-manager RPC. (#12495)
• Move the session-enqueue resource-group accessibility check out of the scheduler repository into the scheduling controller so the repository performs DB reads only and no longer requires a project ID for authorization. (#12807)
• Decouple the session-spec preparer chain from ownership scope by introducing a scope-free SessionResourceSpec, so scheduling that only resolves resources (e.g. a dry-run probe) no longer requires a domain/project/resource-group scope it never uses. (#12849)
• Rebuild the scheduler read path on normalized snapshot loading: user-scoped global quota validation, typed slot value families without ResourceSlot, kernel-agnostic agent selection, and a resource-only compute-schedule fitting check. (#13066)

Resource Group Registration by ID (BEP-1059)

Switched scheduling, fair-share, and usage aggregation onto stable resource group IDs, and stopped agent heartbeats from overwriting the DB-authoritative resource group.

• Switch scheduler-side queries from scaling group name to resource_group_id as a prerequisite for dropping the session's scaling_group_name column (#12655)
• Use stable resource group IDs for fair-share scheduling and usage aggregation. (#12818)
• Keep the manager DB authoritative for an agent's resource group so heartbeats no longer overwrite it; a newly registered agent resolves its configured resource group or falls back to the default one. (#12898)
• Store resource group IDs alongside names in fair-share and usage history records. (#12924)

Reconciler-Based Idle Checking (BEP-1054)

Refined idle-check judgments to carry deadlines and lifecycle phases, decoupled last-access tracking from the legacy checker, and batched the utilization metric queries behind them.

• Maintain session last-access markers in the event dispatcher and stream service unconditionally, decoupling them from the legacy idle.enabled-gated network timeout checker (#12888)
• Return per-checker idle judgments with deadlines and lifecycle phases from the idle-check reconciler. (#12991)
• Read idle-check judgments only from existing session-check assignments. (#13042)
• Add preset-backed batched session utilization metric queries. (#13132)

Scoped App Config (BEP-1052)

Tightened the app config storage model and authorization: a UUID scope column, a single uniqueness constraint, and RBAC/superadmin gates modelled at the action layer.

• Store app_config_fragments.scope_id as a nullable UUID rather than a string, with NULL for the ownerless public scope. (#12984)
• Gate the app config fragment scoped search with scope RBAC by modelling it as a scope action (#13032)
• Gate the app config allow-list, definition and fragment admin searches on the SUPERADMIN role at the action layer. (#13037)
• Key app config fragments with a single UNIQUE NULLS NOT DISTINCT constraint, dropping the partial unique index that kept public rows unique. (#13202)

Action Framework Restructuring

Retyped the action infrastructure onto the shared entity types and wired the pure-ABC per-type validators and monitors into the DI composer.

• Retype the shared action metric and reporter sinks to the new common.entity.types.EntityType, decoupling the action infrastructure from the legacy RBAC EntityType. (#12922)
• Add operation_type() and spec() to the pure-ABC action bases so per-type action monitors can keep emitting the entity:operation keys used by reporter subscriptions and audit logs (#13205)
• Wire the pure-ABC per-type RBAC validators into the shared ActionValidators bundle and the DI composer so entity migrations off BaseAction can enforce RBAC through the new framework (#13206)

Virtual Scope RBAC Ownership Model (BEP-1062)

Consolidated scope creation into a single contract and added covering indexes for large bulk permission checks.

• Add covering indexes to speed up virtual-scope-chain permission resolution on large bulk checks. (#12817)
• Make the RBAC ops layer write virtual-scope membership and scope association together, and let create_scope own the full scope-creation contract. (#13004)

DB Record Retention Management (BEP-1063)

Made retention opt-in by default and folded the legacy error_logs cleanup timer into the retention policy sweep.

• Retire the dedicated error_logs cleanup timer (its rows are now purged by the DB record retention policy sweep) and redefine clear-history as a manual force-sweep plus VACUUM escape hatch over the retention policy layer. (#12963)
• Make DB record retention opt-in by seeding all retention policy categories as disabled by default. (#12996)

Other Improvements

• Clean up the single-host TUI installer internals, and fix two install-flow bugs: --non-interactive --mode PACKAGE never starting the install, and an aborted install continuing instead of stopping. (#12242)
• Represent model service start commands as single command strings internally while preserving deprecated list-form inputs. (#12445)
• Delegate pagination total_count computation to polymorphic strategy methods, removing a redundant count query for unpaginated queries. (#12702)
• Store the audit-log acted_as column as a native UUID instead of a string, so its stored type matches the UUID it is exposed and filtered as. triggered_by remains a string (BEP-1058) (#12763)
• Accept the scoped kernel scheduling-history scope as lists of scope items (kernel: [UUIDScope!], session: [UUIDScope!]) instead of a single kernel_id, so the history can also be queried for every kernel a session owns (#13071)
• Add a by-ID domain query filter and a domain-by-ID GraphQL data loader for UUID-keyed domain lookups. (#13124)

Deprecations

• Deprecate the resource group's enforce_spreading_endpoint_replica scheduler option. Nothing reads it anymore — replica spreading is now governed by the replica group's SessionGroup placement policy, and existing values were migrated onto it. The field stays in the schema and will be removed in the next major release. (#13183)

Fixes

Session Lifecycle & Scheduling

Fixed sessions stalling in intermediate states, ignored resource-group session defaults, and batch reservation times being overwritten.

• Re-trigger image pulling for sessions stuck in PREPARING by including PREPARING in CheckPreconditionLifecycleHandler.target_statuses, so a lost pull-completion event no longer leaves the session unable to progress. (#12455)
• Apply resource group default_session_options (handler_options and resource_opts) to enqueued sessions instead of silently ignoring them. (#12559)
• Fix the compute-schedule dry-run reporting no schedulable agents for a resource group that has agents but no pending sessions. (#12879)
• Preserve the reserved start time of batch sessions in a new sessions.requested_starts_at column instead of overwriting it at the RUNNING transition, and evaluate batch reservations against DB time instead of per-server clocks (#13081)
• Return 404 instead of a silent success or a server error when the agent resource-group update targets an unknown agent or resource group (#13102)

Model Serving & Deployment

Fixed model service start commands losing their shell wrapping, unreadable runtime variants, and rolling-update livelocks.

• Fix model service start commands running without a shell when stored model definitions materialized unset fields as explicit nulls (#12665)
• Align runtime variant fixture start_command with the stored string format so fresh installs seed the same data as migrated databases (#12671)
• Fix runtime variant creation storing a JSON null model definition that made created variants unreadable via the v2 API, and backfill existing rows (#13009)
• Fix rolling-update livelocks where old-revision replicas stuck in PENDING blocked scaling convergence and dead old-revision replicas were refilled against the new revision; during a rollout the old revision is now drain-only and capacity it loses hands over to the new revision immediately (#13087)

Reconciler-Based Idle Checking (BEP-1054)

Corrected idle-check expiration timing and made the checker resilient to deleted keypairs and duplicate lifecycle events.

• Fix active app connection tracking to consistently distinguish session and kernel identifiers. (#12973)
• Sweep only idle checks already judged expired. (#13075)
• Defer idle-check expiration until sessions become eligible for judgment. (#13148)
• Preserve the recorded kernel termination reason against duplicate container lifecycle events, and keep the idle checker from skipping the rest of a cycle when a session's keypair has been deleted (#13166)

Storage & Virtual Folders

Fixed vfolder purge and mount validation, the host usage scale, and GPFS quota serialization.

• Guard vfolder purge against in-use folders: reject purging a vfolder that is mounted by a live session/kernel, referenced by an active model-service endpoint, or not in a purgable status, unless an explicit force flag is set. (#12621)
• Fix vfolder host usage percentage returned by vfolder/_/hosts to be on the 0-100 scale again, so storage status indicators in WebUI and FastTrack reflect actual usage instead of always showing "Adequate" (#12634)
• Fix GPFS quota unset failing with GPFSJobFailedError by serializing the quota limit as a plain integer instead of a human-readable BinarySize string. (#12641)
• Reject vfolder mount aliases set to exactly /home/work, which collided with the intrinsic scratch mount and made container creation fail with "Duplicate mount point" (#12969)

Agent & Kernel Runtime

Fixed container creation and device-scoping failures on the agent, and spurious first-sample utilization spikes.

• Fix create_kernel crashing with a JSON serialization error when a kernel is allocated a unified-memory accelerator device (#12649)
• Scope the ATOM and ATOM+ RSD device group to only the devices assigned to a container, so creating a new session no longer resets the device context of other running containers on the same node. (#12721)
• Reflect changed per-user container UID/GID settings in the container's /etc/passwd and /etc/group when starting sessions from committed (customized) images, instead of silently keeping the stale entries baked into the image (#13086)
• Fix spurious first-sample spikes in hook-derived utilization metrics (e.g. cpu_util) by applying current_hook at Metric creation (#13144)

Resource Leaks

Closed native memory, file-descriptor, and asyncio task leaks in long-running components.

• Fix a manager memory leak where the background-task event stream (GET /events/background-task) leaked an asyncio task and a registered event propagator when a client disconnected before the task finished. (#12686)
• Fix app proxy coordinator event handler leak when legacy circuit initialization times out waiting for the proxy worker (#12729)
• Fix a native memory and file-descriptor leak in all long-running components making aiohttp client requests: since aiodns 3.2, aiohttp defaulted to the aiodns/pycares resolver, and every ephemeral client session leaked a c-ares channel (native heap growth plus one /dev/urandom fd per channel, ~85k fds on 16-day-old agents). aiodns/pycares are removed so aiohttp uses its threaded resolver, and the boot-time self-IP lookup now uses the stdlib loop.getaddrinfo(). (#12979)
• Fix a native memory and file-descriptor leak in every long-running service making aiohttp client requests: since aiodns 3.2, aiohttp implicitly defaulted to the aiodns/pycares resolver, leaking one c-ares channel (native heap plus a /dev/urandom fd) per ephemeral client session. All service entrypoints now force aiohttp's threaded resolver at startup. (#12985)

Security

Closed a session-ownership privilege escalation and a keypair webapp login flaw, and stopped logging the Redis password in plaintext.

• Authorize owner_id delegation in session enqueue against the target owner's scope, closing a privilege-escalation hole where any user could create sessions owned by arbitrary users. (#12568)
• Fix a keypair webapp login flaw where knowing an access key alone could authenticate, and stop embedding the secret key in the (now-expiring) sToken (#13095)
• Mask the Redis password in the agent's startup configuration log instead of printing it in plaintext (#13141)

RBAC Roles & Permissions

Repaired role and permission data that earlier migrations left inconsistent.

• Fix resource preset check failing for project members whose membership is recorded only in association_scopes_entities (#12766)
• Backfill the missing admin-page read permission for project and domain admin roles created at runtime, which an earlier migration skipped due to a role-name pattern mismatch. (#12771)
• Stop auto-assigning admin roles to users joining a scope by clearing auto_assign for roles whose name ends with admin, which the system-role backfill had wrongly enabled. (#12809)

Database Migration & Upgrade

Fixed alembic upgrade aborting or crashing on large or long-lived deployments.

• Fix alembic upgrade aborting on large deployments by chunking bulk INSERTs in RBAC backfill migrations so they stay within PostgreSQL's 32767 bind-parameter limit. (#12535)
• Fix alembic upgrade failing when upgrading from older versions, caused by RBAC migration helpers leaking production ORM columns into their frozen table definitions (#12537)
• Fix alembic upgrade crashing with AttributeError when a model-definition JSONB column holds a JSON null (or other non-object) value, by skipping non-dict rows in the stringify model service start_command migration. (#12548)
• Fix the legacy app_configs downgrade failing with a duplicate enum type error (#12887)

Scheduling History

Fixed the kernel scheduling-history query conditions and its authorization scope.

• Point the kernel scheduling-history query conditions at the from_status / to_status columns the table actually defines, instead of the pre-rename from_phase / to_phase names, and add the missing phase string filters to match the session, deployment, and route history conditions. (#12857)
• Authorize the kernel-scoped scheduling-history search through session read so the owner of a kernel can query its scheduling history instead of receiving a permission error (#13003)

Pagination

Made the pagination cursor safe to bind and consistent with each search's ordering column.

• Bind the pagination cursor as a value instead of splicing it into raw SQL, and answer an invalid cursor with a 400 rather than a server error. (#13065)
• Draw the pagination cursor boundary on the column each search is ordered by, and order presets and runtime variants by a readable column instead of a UUID or a non-unique rank. (#13067)

DB Record Retention Management (BEP-1063)

Fixed the retention sweep never running under the default seed and the spurious alembic check failure it caused.

• Fix DB record retention sweep never running under the default policy seed, and repair clear-history and schema oneshot for the retention tables. (#12995)
• Fix the alembic migration check failing with a spurious drop of the retention_policies table by registering RetentionPolicyRow in the models metadata (#13043)

App Proxy

Fixed a stale connection pool in the coordinator and dropped duplicate response headers in the HTTP backend.

• Add pool_pre_ping and pool_recycle to the appproxy-coordinator database engine so a dropped/stale pooled Postgres connection is transparently reconnected at checkout instead of poisoning the pool with SAVEPOINT can only be used in transaction blocks errors until a restart. (#12507)
• Fix the app-proxy HTTP backend dropping all but the last duplicate response header (e.g. multiple Set-Cookie), which prevented cookie-authenticated apps such as RStudio Server from ever signing in through the proxy (ERR_TOO_MANY_REDIRECTS). (#13194)

TUI Installer

Fixed PACKAGE-mode installs leaving the deployment half-configured instead of aborting.

• Fix PACKAGE-mode TUI installs leaving the default scaling group pointed at the placeholder wsproxy address (:5050) instead of the app-proxy coordinator, and make failing manager CLI steps abort the install instead of being silently ignored. (#12872)
• Fix PACKAGE-mode TUI installs leaving the app-proxy database without any tables (the schema migration silently failed with ModuleNotFoundError); the migration now runs through the coordinator CLI's schema oneshot, and a failing migration aborts the install. (#12875)

Other Fixes

• Fix session download and session ssh failing with HTTP 400 by sending the file list as query parameters to match the manager endpoint contract. (#12509)
• Fix web server login, logout, edu-launcher token login, and no-auth password reset failing when the first configured Manager endpoint is down by routing them through the healthy-endpoint pool instead of pinning to api.endpoint[0]. (#12543)
• Retain the system announcement message when disabling it: disabling now only hides the announcement instead of deleting the stored message, so re-enabling no longer requires retyping. Enabling still requires a non-empty message, and passing an explicit empty message clears the stored text. (#12679)
• Fix tuple-typed config examples (agent port-range, app-proxy worker port ranges and api_port_pool) being rendered as quoted strings in the generated sample.toml files, which failed validation when copied into an actual configuration; they are now emitted as proper TOML arrays. (#12747)
• Return creation timestamps for user and project resource policies in v2 API responses. (#12791)
• Fix user, project and domain usage buckets being over-reported by the number of concurrently running kernels, and rebuild the affected buckets from the recorded per-kernel usage. (#12965)
• Fix the backport workflow silently dropping backports for commits whose message contains a quote character: the milestones result JSON (and author/label values) are now passed to the shell via environment variables instead of inline template interpolation. (#13021)
• Fix admin GQL logging to classify expected 4xx errors from async resolvers at debug level instead of ERROR, and include extensions.code in their error responses (#13129)
• Answer the app config fragment by-names GraphQL queries position by position, holding a name with no fragment at the scope as a null instead of dropping it. (#13211)

Documentation Updates

• Add BEP-1056 for scope-linked metric catalog and dashboards. (#12531)
• Add BEP-1055 defining the scheduler mechanics for preempting low-priority running sessions (terminate/reschedule). (#12564)
• Add BEP-1057 (Agent Re-architecture) proposal. (#12565)
• Add BEP-1058 proposing super-admin user impersonation: execute under a target user's permission context while auditing both the trigger and effective identities. (#12653)
• Add BEP-1059 proposing to make the manager DB the source of truth for agent↔resource group registration. (#12703)
• Add BEP-1060 defining the convention for v2 connection-type (to-many) nested filters (some/every/none), with its first application to filtering deployments by replica state. (#12730)
• Change the alembic revision version comment convention to use the NEXT_RELEASE_VERSION placeholder instead of a hardcoded version. (#12770)
• Add BEP-1061 proposing a structured Agent kernel lifecycle (prepare/create/terminate) with async triggers and transition events reported to the manager, keeping the authoritative kernel status state machine in sokovan. (#12783)
• Add BEP-1062 proposing the Virtual Scope RBAC ownership model that replaces the recursive scope-walk with a non-recursive two-hop scope -> virtual_scope -> entity resolution, documenting the as-built tables/ops and the migration plan. (#12842)
• Update BEP-1054 to define DB-backed idle-check deadlines, persisted checker results, and separate refresh and expiry-sweep stages. (#12900)
• Amend BEP-1055 to preempt on a scope-local job_priority axis (restricted to the same owner) instead of the global scheduler priority. (#12939)
• Add BEP-1063 proposing a general-purpose DB record retention management layer where a super-admin sets a retention period per data category and the manager periodically purges older records via a batched, referentially-safe LeaderCron sweep. (#12940)
• Clarify the alembic README head-reconciliation rule: repoint your own unmerged migration, but add an empty merge migration when main itself has multiple released heads. (#12983)
• Amend BEP-1054 to separate idle checking into assignment sync, initial grace period, judgment, and expiry-sweep stages with distinct reconciliation histories. (#13022)
• Add BEP-1065 specifying at-rest encryption of keypair secret keys with a self-describing stored format and key-version rotation. (#13164)
• Add BEP-1064 proposing SessionGroup placement, where a group of related sessions carries an agent-level policy to spread or pack its members, applied ahead of the resource group's own agent selection strategy. (#13165)
• Add BEP-1072 proposing release branch isolation and backport automation: cut the version branch from the rc1 release on main, decide backport targets by PR title prefix and labels, leave failed backports as draft PRs, and split the changelog into per-version-branch files. (#13210)

External Dependency Updates

• Update bssh and bssh-server binaries to v2.2.3 (#12558)

Miscellaneous

• Move BaseFilterAdapter from the manager data layer to the repositories layer to remove a data-to-models dependency edge. (#12640)
• Fix a typecheck regression where the sequencer test omitted the now-required resource_group_id when constructing SessionWorkload. (#12697)
• Remove the "resolved via DataLoader" implementation detail from GraphQL nested-field descriptions so the schema documents the interface, not how values are fetched. (#12726)
• Lower per-execution agent logs kernel.execute and CodeRunner.attach_output_queue from INFO to DEBUG to reduce log noise. (#12777)
• Stop granting RBAC permissions on the app_config entity type and remove the rows already granted; the entity it guarded was removed with the legacy AppConfig layer, and its replacement is not RBAC-guarded. (#12839)
• Type the single-entity, bulk, and scope base action identifiers as EntityID. (#12897)
• Enforce the manager layer dependency order (repositories must not import services/api, services must not import api) at build time via Pants visibility rules. (#12932)
• Reconcile the two diverged alembic heads (b988f01b17a1, a1c4e7b93f22) by adding an empty merge migration (577c7a215934) instead of editing the released migrations' down_revision. (#12978)
• Run the full CI test suite on release PRs (detected by the release: PR title prefix) so that release-blocking failures surface before the tag-push release CI, and warn when the VERSION file is modified in a non-release PR (#13036)
• Merge diverged manager Alembic migration heads. (#13094)

Test Updates

• Register ImageRow in the RBAC validator and bulk role-permission test ORM clusters so the SQLAlchemy mapper registry can resolve KernelRow's relationship("ImageRow"). (#12533)
• Register ImageRow in the scaling group service test conftest so the CRUD tests pass when the target is run in isolation (the string-based KernelRow.image_row relationship needs ImageRow in the SQLAlchemy registry at mapper-configuration time). (#12631)
• Seed association_scopes_entities in resource preset check_presets repository tests to match project membership resolution. (#12813)
• Add component tests for the kernel scheduling-history v2 SDK client. (#12997)
• Add component tests for the replica-group scheduling-history REST v2 endpoints. (#13061)
• Cover the scoped app config fragment upsert against a real database, including the RBAC scope binding and the on-conflict behaviour at every scope type. (#13122)
• Provision project virtual scopes in component test fixtures. (#13139)
• Run the unit-test Postgres fixture on 16.3, the version the product requires, instead of 13.6. (#13200)

Full Changelog

Check out the full changelog until this release (26.8.0rc1).

Full Commit Logs

Check out the full commit logs between release (26.7.0) and (26.8.0rc1).