Skip to content

v0.2.7

Choose a tag to compare

View all tags
@Lincyaw Lincyaw released this 09 Mar 01:15
· 296 commits to main since this release
v0.2.7
49280fb

What's Changed

🔧 Coalesced Pool Scaling (fixes concurrent 500 errors)

Replaced the per-request read-modify-write pattern in ensureCapacity with a per-pool coalesce goroutine. All concurrent scale-up requests for the same pool are batched within a 50ms window and executed as a single K8s Update (with RetryOnConflict fallback).

Before: 512 concurrent sessions → 511 K8s conflict errors (500s returned to client)
After: 512 concurrent sessions → 1-2 K8s API calls, zero conflicts

Also fixed a race condition in createPool where concurrent goroutines could split the sessionCount across different poolState objects when pool creation failed.

🚀 maxReplicas Eager Scaling Hint

New optional field on managed session creation that tells the server to eagerly scale the pool up to the specified target on the first request, instead of scaling incrementally.

# Server scales pool directly to 8 replicas on first request
with ManagedSession(
    image="python:3.11-slim",
    experiment_id="my-exp",
    max_replicas=8,
) as session:
    ...
  • maxReplicas is an eager target, not a hard ceiling — if actual demand exceeds it, the pool continues to scale up
  • Multiple callers with different hints: the highest value wins (e.g., one caller says 8, another says 16 → pool scales to 16)
  • Available in both Python SDK (ManagedSession(max_replicas=...)) and REST API (POST /v1/managed/sessions body field maxReplicas)

🩹 Gateway & Operator OOM Fix

Both Gateway and Operator were repeatedly OOM-killed (exit code 137) under load:

  • Gateway: 5 restarts, 256Mi limit with 191Mi usage at idle
  • Operator: 7 restarts, 256Mi limit with 178Mi usage at idle

Resource changes (Helm defaults):

Component Before After
Gateway memory 256Mi / 128Mi 8Gi / 2Gi
Gateway CPU 500m / 100m 4 / 1
Operator memory 256Mi / 128Mi 8Gi / 2Gi
Operator CPU 500m / 100m 4 / 1
managedPool.maxReplicas 50 100

📊 Dashboard Query Optimization

Grafana dashboard queries were extremely slow with 200+ managed pools. Root cause: histogram queries with {pool=~"$pool"} (default .*) expanded to pools × buckets (2000+) time series per panel.

Fixes:

  • Removed per-pool filter from all histogram_quantile panels — they already sum by (le) so the pool filter only added regex overhead
  • Dropped step_type grouping from Step Duration/Result panels (unique step names cause cardinality explosion)
  • Added max_over_time smoothing and interval hints for Top 20 Pools panel
  • Added minimum evaluation intervals to reduce Prometheus load

Full Changelog: v0.2.6...v0.2.7

Assets 2
Loading