Release v0.13.0rc1

SkyPilot v0.13.0rc1: Sky Batch, Hugging Face Storage, Lifecycle Hooks, Cost Caps, Blackwell GPUs, and More

SkyPilot v0.13.0rc1 introduces Sky Batch for large-scale batch inference and data processing across worker pools, a native Hugging Face storage backend (hf://), a generalized lifecycle-hooks framework for running custom logic around cluster events, and the **sky debug-dump** command for one-shot troubleshooting. This release also adds per-instance cost caps, NVIDIA Blackwell / CUDA 13 images, and GKE Autopilot support.

Get it now with:

uv pip install "skypilot[all]==0.13.0rc1"

Or, upgrade your team SkyPilot API server:

NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.13.0rc1

helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --set apiService.image=berkeleyskypilot/skypilot:$VERSION \
  --version $VERSION --devel --reuse-values

Deprecations & Breaking Changes:

• **api_access renamed to api_server_access, now defaulting to True** (#9169). Tasks now automatically receive API server credentials. The legacy api_access key is still accepted for backward compatibility.
• **sky.jobs.queue(version=1) remains deprecated** and continues to emit a warning; use sky.jobs.queue(version=2), which returns richer job metadata as dictionaries (#9118).
• Removed all_includes_in_cluster from the SkyPilot config schema (#9671).

Highlights

[New] Sky Batch

**sky batch** brings large-scale batch inference and data processing to SkyPilot with a simple Python API (#8850, #9438, #9456). Given a dataset and a processing function, Sky Batch splits the data into batches, distributes them across a pool of GPU/CPU workers, and collects the results, with no manual sharding, provisioning, or failure handling. Workers are reused across jobs, so expensive setup (installing packages, downloading weights, loading models onto GPUs) happens only once.

import sky

# Point at your input data in cloud storage
ds = sky.batch.Dataset(source='s3://my-bucket/prompts/')

# Map a processing function over the dataset across a worker pool
ds.map(my_inference_fn, pool='my-gpu-pool', output='s3://my-bucket/results/')

Inputs and outputs can be read from and written to Amazon S3 (s3://) or Google Cloud Storage (gs://).

[New] Hugging Face Storage Backend

SkyPilot now supports Hugging Face as a first-class storage backend via the hf:// scheme (#9698, #9763). It wraps read-write Hugging Face Buckets (Xet-backed, S3-like object storage) and, through the same handle, read-only mounting of model, dataset, and space repos. Bucket lifecycle operations go through huggingface_hub, and MOUNT/MOUNT_CACHED modes use the FUSE-based hf-mount, consistent with SkyPilot's existing mount infrastructure.

file_mounts:
  # Read-write Hugging Face Bucket
  /data:
    source: hf://buckets/my-org/my-bucket
  # Read-only model / dataset / space repo
  /model:
    source: hf://meta-llama/Llama-3.1-8B
    mode: MOUNT

URL formats: hf://buckets/<ns>/<bucket> (bucket, read-write), hf://<ns>/<model>[@<rev>], hf://datasets/<ns>/<ds>, and hf://spaces/<ns>/<sp> (repos, read-only). Authentication reads HF_TOKEN from the environment or the Hugging Face CLI token file.

[New] Lifecycle Hooks Framework

The autostop hook introduced in v0.12 has grown into a generalized lifecycle-hooks framework via resources.hooks (#9064). Each hook declares a script and the events it fires on: autostop, preemption, and down. Use it to checkpoint state, sync experiment trackers, or send notifications around cluster lifecycle transitions.

resources:
  hooks:
    - run: |
        wandb sync
        curl -X POST $SLACK_WEBHOOK -d '{"text": "Cluster shutting down"}'
      events: [autostop, down]
      timeout: 300

The legacy autostop.hook YAML continues to work with a deprecation warning.

[New] sky debug-dump: One-Shot Troubleshooting

The new **sky debug-dump** command collects diagnostic information (logs, state, and configuration) for clusters, requests, and managed jobs into a single zip for troubleshooting and bug reports (#8868, #9752, #9753). Dumps are stored under ~/.sky/debug_dumps/ and automatically cross-link related resources (e.g. a cluster and its associated requests) so you capture complete context in one command.

sky debug-dump --cluster my-cluster

Cost Caps: Max Hourly Cost

You can now set a per-instance hourly budget cap with resources.max_hourly_cost (#9132, #9170). Only instances at or below the limit are considered during optimization. When use_spot is set, the cap applies against spot prices; otherwise against on-demand prices.

resources:
  accelerators: A100
  max_hourly_cost: 10.0

NVIDIA Blackwell & CUDA 13 Images

SkyPilot's default GPU image now ships NVIDIA driver 580 (open) + CUDA 13 for Blackwell support (#9639), with a legacy CUDA 12 image retained for pre-Turing GPUs and an arm64 build fix (#9789). RunPod adds RTX PRO Blackwell GPUs to its catalog (#9187).

GKE Autopilot Support

SkyPilot now detects GKE Autopilot clusters and trusts Node Auto-Provisioning (NAP) instead of walking existing node pools, so GPU requests provision correctly on Autopilot instead of being incorrectly rejected (#9565).

---

What's new: compared to v0.12.0

API Server

• RBAC viewer role for read-only dashboard/API access (#9681).
• Reconcile internal daemon rows on API server boot (#9668) and skip request env overlay for internal daemon requests (#9663).
• Bundle SSH credentials with the launch response (#9586).
• Refinements for graceful upgrade (#9317) and MIN_COMPATIBLE_API_VERSION bump for 0.12 clients (#9121).
• Reap multiprocess Prometheus files from dead/exited workers (#9689, #9690) and periodic scrape-timeout tuning (#9480, #9434).
• Return 404 for missing dashboard _next/ assets instead of index.html (#9722); don't cache error responses on /dashboard/_next paths (#9498).
• Fixed incorrect headers passed by oauth2-proxy (#9468) and oauth pod scheduling (#9288).
• Async engine improvements: hand asyncpg the libpq DSN via async_creator (#9483), put_async for the queue backend (#9346).
• Workspace controls: name-length limits (#9162), workspace permission checks for enabled_clouds and sky check (#9284), cached permission checks (#9125).
• Helm chart: devMode flag for in-cluster iteration (#9609), persist the whole sky_logs dir (#9454), optional server log mirror (#9754), configurable probes (#9477), disable session affinity on ingress (#9347), option to disable serving the server log (#9140).

Kubernetes

• GKE Autopilot support via autopilot.enabled detection and NAP trust (#9565).
• Per-workspace namespaces via context_configs (#9640).
• Container operations for clusters on Kubernetes (#9066).
• **ephemeral-storage requests/limits** on pods (#9053), allowed_nodes config for node-level filtering (#9151), pod spread constraints (#9296).
• Configurable APT mirrors (#9521, #9553) and timeouts on APT commands (#9375).
• Surface pod OOM and eviction reasons for jobs and clusters (#9758); better provisioning-failure reasons (#9389); fix misleading "Pulling image" status during init-container execution (#9702).
• Treat pods as scheduled on binding rather than container status (#9766); auto-increase provision timeout when an autoscaler is configured (#9322).
• Respect configured tolerations in node-health display (#9750).
• Allow pod_config to override runtimeClassName for GPU pods (#9241); probe Ray ports when hostNetwork: true (#9644).
• Fail fast when a cluster has no default StorageClass (#9713); auto-clamp CPU/memory for capacity (#9179).
• network_tier: best for OCI OKE RoCE (#9622); allow excluding the in-cluster context from allowed_contexts: all (#9637).
• Fixed GPU labeller crash preserving method metadata (#9152), volumeMounts patch-merge key from name to mountPath (#9358), write permissions for all volume mounts at startup (#9221), owner-identity check for scoped kubeconfig names (#9725, #9190). Updated default/base k8s images (#9618, #9590).

Volumes

• Volume auto-mounts on Kubernetes (#9205, #9353).
• New k8s-hostpath volume type for node-local cache (#9165).
• Volume detail page with saved/shown volume YAML (#9101) and refreshed volume page styling (#9122).
• Infer access_mode from an existing PVC for use_existing volumes (#9359); check the existing backend when creating volumes (#9104); add volume mount check (#9343).
• Show Ti units for volume sizes ≥ 1024Gi (#9369); don't rewrite region to in-cluster when in-cluster auth is unavailable (#9743).

Storage

• Hugging Face storage backend (hf://) for buckets and repos (#9698, #9763).
• Read-only support for MOUNT (#9225).
• Upload v2 improvements: narrowed locking (#9460), skip-assemble support (#9458), delayed extract (#9376), and upload chunk size capped to fit Cloudflare's 100MB limit (#9360).
• Upgraded goofys to 0.24.0 (#9222); mount-command cleanup for S3CompatibleStore (#9080); fixed git "dubious ownership" during workdir sync (#9352).

Core & Backend

• Support specifying both a cloud VM image and a Docker image on the same resources (#9759).
• Deep-merge workspace config with global cloud config (#9788).
• Speed up log download and tailing for large jobs (#9464).
• Turn silent launch misconfigurations into explicit errors (#9423) with improved user-facing messages (#9403); detect file paths mistakenly used as cluster names (#9233).
• Fall back to SSH when the skylet gRPC endpoint is unreachable (#9287); cancel in-flight skylet gRPC calls on context cancel (#9620); bind gRPC to all interfaces (#9282).
• Bumped default cluster event retention to 30 days (#9462).
• Deterministic config hash (#9415), reproducible wheel builds (#9368), and excluding __pycache__ from the built wheel (#9413).
• Allow relative file_mounts destinations again (#9537); fixed cluster re-provisioning and atomic SSH-key writes (#9476); prioritize user SSH keys over cluster keys in Host * config (#9099).
• Fixed autostop ignoring active SSH after stop/start (#9539); close /dev/fuse fd after passing it to the client (#9463); fixed a flock leak in the uv_spawn child process (#9431); fixed browser opening on WSL for sky api login (#9178).
• Moved the catalog dir to SKY_RUNTIME_DIR and removed import-time filesystem side effects (#9605); let packages expose client SDKs under the sky namespace (#9784).
• Expanded plugin extension points and config sections for third-party extensibility (#9494, #9510, #9457, #9616).

Managed Jobs & Serve

• **--tail for sky jobs logs** (#9449), preload_content for sky.jobs.tail_logs (#9662), and a wait helper in the programmatic SDK (#9138).
• Provisioning spinner and dashboard links during sky jobs launch (#9761, #9135, #9191); optionally merge cluster launch-progress events into job events (#9804).
• Custom job recovery strategy configuration (#9154); retry transient DB errors in the controller monitor loop (#9748); daemon to sweep expired managed-job API access tokens (#9588).
• Fixed git commit metadata not captured in managed jobs (#9184) and git workdir env vars not set for JobGroup / multi-task DAGs (#9251); fixed user-hash races in concurrent job-group launches (#9248).
• Serve: ServiceStatusRunner extension point (#9801), new user-configurable options (#9116), and improved max-services error messages (#8897).
• Consolidation mode: use short workers (#9442) and skip the two-hop path (#9444).

Pools & Batch

• Sky Batch for distributed batch inference over worker pools (#8850, #9438, #9456).
• Prioritize idle workers for scale-down (#9098); speed up the pool dashboard with batched queries and indexes (#9636); fixed pool consolidation under deploy-mode auto-enable (#9342).

Infrastructure & Cloud Integrations

AWS

• Support for aws login (2026 AWS CLI command) (#9526) and subnet selection (#9127).
• Add tags to security groups for scoped IAM policies (#8981); fixed false MISMATCH error when subnet_names span multiple AZs (#9791); added AZ-fetch timeouts to prevent sky check hangs (#9244, #9240).

GCP

• Select specific VPC subnets (#9334); set provisioningModel=RESERVATION_BOUND for DENSE reservations (#9779); added missing required instance types to the catalog (#9472); pinned/bumped the gcloud SDK (#9587, #9535); sped up sky check (#9095).

Nebius

• Native security groups with open_ports replacing UFW (#9563); use managed disks to create instances (#9432); parallelize multi-node VM creation (#9570); custom image_id (#9133); fixed autodown (#9573).

Slurm

• cpu_partition config for CPU-only tasks (#9281) and gpu_partition_map for GRES without a GPU type (#9237).
• Let users override --time and fall back to DefaultTime when MaxTime is UNLIMITED (#9633); preserve SLURM_CONF* env vars when launching srun (#9425).
• Fixed a proctrack barrier hang on multi-node container jobs (#9181), fractional memory truncation in sbatch (#9174), numeric partition names treated as integers (#9168), and --mem when memory is not consumable (#9171).

RunPod / Vast / PrimeIntellect

• RunPod: RTX PRO Blackwell GPUs in the GPU name map (#9187); pass formatted image name to create_template (#9550).
• Vast: fixed API compatibility with vastai-sdk v6+ (#9314); made account-level SSH key registration best-effort (#9631).
• PrimeIntellect: fixed query_instances TypeError on retry (#9259) and instance update after _get_instance_info (#9327).

SSH Node Pools

• Install Prometheus + node-exporter in sky ssh up (#9402).
• Auto-enable HA K3s for pools of 3+ nodes (#9351).
• Wait for async Ray setup on SSH node pools (#9299); support redirect in the WebSocket SSH proxy (#9167).

User Experience

Dashboard

• Batch-edit user roles and workspace allowed_users (#9696); show counts above the users and service-account tables (#9655).
• Admin-configurable external links (#9603) and detection of W&B Dedicated Cloud run URLs (#9383).
• Provision log in the cluster detail page (#9201, #9344).
• Managed Jobs filter polish: More dropdown, Active/All toggle, default to "Mine" (#9707), default sort by ID descending (#9522), and don't block the table on /status when the controller is up (#9645).
• Unified filter-bar sizing and styling across list pages (#9407); improved performance for cluster detail and job pages (#9195); quieter upgrade banner (#9654).
• Fixed infinite-spinner states on root and detail pages (#9139, #9147), routing for catch-all pages opened in new tabs (#9137), and layout flashes on slow plugin loads (#9208).

CLI

• **sky api login --no-browser** for headless environments, continuing to poll when a browser can't be opened (#9677, #9676).
• **sky debug-dump** troubleshooting command (#8868).
• Display autostop in hours when minutes are divisible by 60 (#9453); JSON output for cost-report (#9504).

Dependencies

• Bumped protobuf to 5.29.6 and black to 24.10.0 to close CVE alerts (#9541).
• Pinned kubernetes < 36 due to an RBAC regression, while accepting dict[K, V] openapi_types for kubernetes >= 36 (#9685, #9682).
• Capped azure-cli < 2.87.0 (azure-mgmt-storage 25.0.0 breaks Azure storage) (#9774).
• Dropped the pyopenssl upper limit (#8070); pinned click < 8.3.0 in relevant runtime installs (#9459, #9763).

Documentation & Examples

• Sky Batch docs including custom formats (#9438); MOUNT_CACHED config documentation (#9130); api_server_access documentation (#9115); refreshed managed jobs, quickstart, and agent-skills docs (#9564, #9250).
• New examples: autonomous code optimization (#9182), parallel autoresearch (#9092, #9131), RL post-training with policy pushback (#9625), Cosmos3 fine-tuning and nano (#9809, #9790).
• "Copy page as Markdown" button for LLM-friendly content sharing (#9093); faster incremental docs builds (#9396); added Slurm to the CoreWeave installation guide (#9674); refreshed cloud logos (#9642, #9602).

Testing & CI/CD

• Nightly workflow: run once daily at 00:00 UTC and drop the preflight schedule (#9739); trigger a downstream workflow after PyPI publish (#9738).
• Fixed a race condition in the release workflow that caused untagged releases (#9227).
• Dropped Black in favor of yapf+isort for the IBM provider (#9557); numerous test reliability and flakiness fixes.

Contributors

Thank you to all contributors who made this release possible!

@aylei, @Michaelvll, @DanielZhangQD, @zpoint, @kevinmingtarja, @lloyd-brown, @concretevitamin, @cg505, @SeungjinYang, @rohansonecha, @kevinzwang, @kyuds, @romilbhardwaj, @alex000kim, @andylizf, @cblmemo, @hentt30, @williamsnell, @Jobarion, and many others.

Note: this is a draft. Use GitHub's "Generate release notes" on the tagged release to finalize the exact contributor @Handles and the "New Contributors" section.

Full Changelog

For a complete list of changes, see the commit history.