Skip to content

SkyPilot v0.10.2

Choose a tag to compare

View all tags
@Michaelvll Michaelvll released this 27 Aug 08:04
v0.10.2
fc35b36
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

SkyPilot v0.10.2: Enhanced observability, programmatic SDK, performance, and more

SkyPilot v0.10.2 brings values in cluster management, improved Kubernetes support, programmatic SDK, and numerous stability and performance enhancements for production use from teams with a large number of workloads.

Get it now with:

uv pip install "skypilot>=0.10.2"

Highlights

Cluster observability improvements

Get provisioning logs with the new --provision option for sky logs (#6638):

sky logs --provision <cluster-name>

Find the detailed reason for your cluster failure, e.g., OOM, with the cluster events (#6590, #6593, #6615, #6620, #6621, #6667, #6658, #6609, #6617):

SDK: Process your logs while streaming

SkyPilot introduces a new preload_content option for tail_logs to enable processing logs while streaming.

logs = sky.tail_logs(cluster_name, job_id, follow=True, preload_content=False)
for line in logs:
    if line is not None:
        if 'needle in the haystack' in line:
            print("found it!")
            break
logs.close()

Performance Improvements

  • Job dashboard page loading speed improvement for 10k+ jobs (#6714, #6652)
  • Volume mounting on Kubernetes from 2mins to seconds (#6679)
  • Increase the connection limit from a single client: 10 to 100 (#6782)
  • Speed up sky down for AWS clusters with exposed ports by 4x (#6629, #6663, #6720)
  • Fix the OOM issue with large file_mounts (#6865)
  • Enhanced SSH connection robustness (#6715)

What's New

Infrastructure-specific Improvements

  • Kubernetes:
    • Label support for volumes (#6696)
    • Better authentication handling for in-cluster services (#6600)
    • Proxy setup guide for Kubernetes (#6618)
    • Improved error handling: Enhanced failover error messages (#6613)
    • Enable write access to conda base environment (#6766)
    • Better GPU label detection with empty labels (#6859)
  • AWS:
    • Customizable SSH users (#6625)
    • Root device name detection (#6644)
    • Support Rocky Linux (#6711)
    • Fix fractional GPU quantity for P5.48xlarge (#6722)
  • Nebius:
    • Memory configuration support (#6832): Users can now properly specify memory requirements (e.g., 64GB instances)
    • More robust cluster status fetching (#6693, #6672)
    • Enhanced VRAM calculation (#6628)
    • Improved infinite waiting fixes (#6674)
  • RunPod:
    • Docker image handling with any_of config (#6728)
    • Docker name conflict resolution (#6769)
  • Lambda: Support B200 on Lambda (#6816)
  • R2 mount issue resolution (#6662)

Dashboard Enhancements

  • Managed Jobs:
  • User Management:
    • Allow editing allowed_users and private settings for workspaces (#6566)
  • Miscs:
    • Fixed tour auto-start behavior (#6654)

API Server Enhancements

  • Robust jobs/serve controller against upgrades (#6779)
  • Prometheus metrics in deployment mode (#6712)
  • Docs for using Cloud SQL for states (#6587)
  • State management improvements:
    • Unique constraint violation detection for PostgreSQL (#6773)
    • SQLAlchemy warning suppression (#6796)
    • Reduce DB creation operations (#6594)
  • Admin Policy & Permissions & RBAC:
    • Apply admin policy for volumes (#6668, #6781, #6749)
    • Enhanced restful admin policy in testing (#6353, #6626)
    • Backward compatibility handling for admin policies (#6692)
    • Optimized permission module logging (#6805)

UX/API Improvements

  • CLI Enhancements:
    • Better managed job log tailing behavior (#6719)
    • Improved --config option documentation (#6794)
    • Better logs organization for downloading (#6795)
    • Warning for disk size specifications on Kubernetes (#6637, #6684)
    • Client version and commit info in sky api info (#6748)
    • Improved cluster event messaging and wording (#6686, #6688)
    • Better user feedback with spinner for delay messages (#6575)
    • Enhanced pod configuration validation (#6825)
  • SDK Improvements:
    • Improved transient failure handling (#6808, #6807): More intelligent retry mechanisms
    • Simpler log streaming utility (#6750): More intuitive API for processing logs in real-time with iterator-based approach
    • Better response typing (#6659, #6718, #6527)
    • Exposed sky.endpoints function (#6599)
    • Enhanced decoder backward compatibility (#6810)
    • Comprehensive endpoints documentation (#6815)
  • Robustness and Performance Improvements:
    • Fixed authentication with simplejson (#6698)
    • Reduced import statement overhead (#6641, #6645, #6648)
    • Optimized directory utilities (#6646)
    • Resolved API logging issues (#6619)
    • Enhanced retry logic for non-transient errors (#6844)
    • Better handling of missing pods with termination filtering (#6697)
    • Fixed various cluster name querying issues (#6616, #6624)
    • Improved job import handling in sky init (#6623)

Testing and Infrastructure

  • Test Improvements:
    • Automatic retry for core tests (#6710)
    • Expanded nightly build coverage (#6695)
    • Staggered test execution (#6732)
    • Remote server test support (#6819, #6823)
    • New smoke tests for provisioning logs (#6790, #6818)
    • Fixed endpoint comparison in unittests (#6578)
    • Fixed TPU test example failure (#6673)
    • Added Kubernetes volume merging unit tests (#6813)
    • Added support for custom cloud config files in smoke tests (#6851)
    • Enhanced serve status checks after termination (#6713)
    • Include Nebius VMs tests in CI/CD (#6835)
  • CI/CD Enhancements:
    • PAT token for release publishing (#6612)
    • Helm lint in pre-commit hooks (#6798, #6803)
    • Fixed nightly build triggers (#6610)
  • Helm Charts & Deployment:
    • Fixed RBAC template rendering issues (#6737, #6742)
    • Resolved ingress configuration for oauth in grafana (#6799)
    • Enhanced values.yaml documentation (#6800)
    • Better patch strategy for merging Kubernetes configs (#6653)

Other feature improvements

  • SSH Node Pool: region-specific launches (#6767)
  • SkyServe: GPU-aware Load Balancing (#6147)
    SkyServe now intelligently scale based on GPU types. Set different QPS targets across heterogeneous GPUs: 
    load_balancing_policy: instance_aware_least_load
replica_policy:
  target_qps_per_replica:
    "H100:1": 2.5    # H100 can handle 2.5 QPS
    "A100:1": 1.25   # A100 can handle 1.25 QPS
    "A10:1": 0.5     # A10 can handle 0.5 QPS

Backend

Examples and Documentation

Contributors

We thank all contributors who made this release possible!

New Contributors: @miltava, @tomzx, @webconn, @hongsu-moreh, @ibpark-moreh, @nathan-liner

All Contributors: @DanielZhangQD, @cblmemo, @kyuds, @SeungjinYang, @lloyd-brown, @romilbhardwaj, @rohansonecha, @zpoint, @aylei, @SalikovAlex, @Michaelvll, @kevinmingtarja, @miltava, @Maknee, @cg505, @tomzx, @concretevitamin, @webconn, @sethkimmel3, @andylizf, @hongsu-moreh, @ibpark-moreh, @lucamanolache, @nathan-liner

Special thanks to the community for bug reports, feature requests, and pull requests that helped improve SkyPilot!

Full Changelog

For a complete list of changes, see the commit history.

Contributors

tomzx, concretevitamin, and 21 other contributors
Assets 2
Loading