Skip to content

v2.9.1: Arena chat sessions & PAT auth, MoE LoRA under ZeRO-3, GRPO deadlock fix

Choose a tag to compare

@jaimesabalbermudez jaimesabalbermudez released this 12 Aug 13:12
· 62 commits to main since this release
814f138

Features

  • Arena chat sessions: LLM deployments now keep conversation state server-side. Agent.list_sessions /
    get_session / delete_session, an optional session_id on generate() and generate_stream(), and a new
    arena agent sessions list/get/resume/clear/delete command group. The CLI keeps one conversation going per
    deployment, with --new-session to start fresh and --session-id for a one-off; resume with no id opens an
    arrow-key picker built on termios and msvcrt, so it adds no dependency (#661).
  • PAT-authenticated inference: Agent takes a single credential, sent as Authorization: Bearer on every
    route and falling back to ARENA_API_KEY. It is never logged, shown in repr(), or written to disk (#661).
  • memory_scope: deploy_agent and arena agent deploy take an optional --memory-scope user|organization, omitted from the request unless set so a redeploy keeps the stored scope (#661).

Fixes

  • Expert LoRA attaches under ZeRO-3 without all-gathering the packed experts. get_peft_model and
    upgrade_moe_param_wrappers read only the targeted parameters' shapes, dtype and device, so partitioned params
    now get zero-storage ds_shape views. The gather OOMed at attach time on large MoEs, around 55 GB of experts per
    rank for Nemotron-3.5-Lightning-30B on 80 GB A100s (#658).
  • ZeRO-3 persistent params stay resident while the fetch trace is incomplete. deepspeed 0.19.3 honours
    ds_persist in release_sub_module only once the trace completes, and leaf-module models never complete one, so
    every sub-threshold parameter was re-gathered on each use despite being marked persistent (#658).
  • Multi-process GRPO no longer deadlocks on advantage filtering. Per-rank sample dropping desynchronized the
    data-parallel collective schedule, parking a fully filtered rank at a barrier until the NCCL watchdog fired.
    Multi-process runs now keep the full batch and zero the advantages of filtered samples; single-process runs keep
    the drop and its compute saving (#660).
  • A stale deployment URL recovers instead of failing with a bare 404. Redeploying moves a deployment, so
    open_inference_agent catches the 404 at bind time, refetches the binding once, and retries, repairing the cache
    for later commands. Narrow by design: only a 404, only when the cached URL was used, and only when the refetched
    URL differs (#661).
  • Family-detection tests use the released NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 id in place of the dev
    preview name. The Hub redirects the old id, so nothing was broken; detect_model_families matches on the
    nemotron substring and dispatch is unchanged (#663).

Breaking Changes

  • Arena inference callers authenticate as themselves with a user PAT. Deployment API keys are gone from the platform, and
    the deployment api_key is gone from the binding cache: save_binding purges any key an older release left in
    ~/.arena/inference.json, so one write cleans every stale entry (#661).

Versions

agilerl-arena==0.2.0 released alongside, carrying chat sessions and PAT-based inference auth; the arena extra pins it.

What's Changed

  • Attach expert LoRA under ZeRO-3 without gathering the packed experts by @micdoh in #658
  • Add PAT-authenticated chat sessions for Arena LLM deployments by @jaimesabalbermudez in #661
  • Mask zero-advantage samples instead of dropping them under multi-process GRPO by @micdoh in #660
  • Use the released Nemotron 3.5 Lightning model id by @micdoh in #663
  • v2.9.1: Arena chat sessions with PAT auth, ZeRO-3 MoE LoRA attach fix by @jaimesabalbermudez in #662

Full Changelog: v2.9.0...v2.9.1