[REFACTOR] Refactor LingBot stateful generation and session lifecycle

[lzx1413]

Description

This PR refactors LingBot-World-Fast stateful chunk generation around explicit generation sessions and worker-local runtime state.

It replaces implicitly pipeline-owned state with externally owned generation sessions and cache handles. Long-lived KV, scheduler, timestep, RNG, and VAE streaming decode state now remains in worker-local cache registries instead of being passed between chunk requests.

The refactor also introduces deterministic session cleanup, transactional poison/release semantics, action-driven single-chunk generation, and stronger isolation between concurrent or sequential sessions.

Four-GPU offline and interactive WebRTC examples, assets, tests, and deployment documentation are included to validate and demonstrate the refactored architecture.

Motivation

LingBot generates video incrementally, with every chunk depending on mutable state produced by previous chunks. The previous implementation coupled this state to pipeline and worker execution details, making ownership, failure recovery, and session isolation difficult to reason
about.

In particular, the runtime needed clearer guarantees for:

• ownership of long-lived generation state
• session isolation across KV and VAE decode caches
• deterministic worker and cache cleanup
• safe behavior after partial generation failures
• action-driven generation without materializing all controls in advance
• multi-GPU offline and interactive streaming execution

This refactor establishes explicit session and cache lifecycle boundaries. Callers own generation sessions, workers own device-local state, and cache handles connect the two without transferring large mutable objects between requests.

Type of Change

• Bug fix (session isolation and failure cleanup)
• New feature (action-driven chunk API and interactive examples)
• Breaking change
• Performance improvement
• Code refactoring
• Documentation update
• Other

Changes Made

Stateful generation architecture

• Refactor LingBot chunk generation around externally owned generation sessions.
• Expose an action-driven single-chunk generation API.
• Keep KV, scheduler, timestep, and RNG state in worker-local cache registries.
• Address worker-local state through lightweight cache handles.
• Avoid transferring long-lived mutable generation state between chunk requests.
• Defer external control materialization until each chunk is generated.

Session and cache lifecycle

• Add explicit cache creation, lookup, poison, release, and shutdown semantics.
• Make failed state unusable so partially updated caches cannot be reused.
• Ensure worker shutdown deterministically releases session-owned resources.
• Harden session creation and cleanup against partial initialization failures.
• Prevent stale sessions from affecting later generation requests.

VAE streaming isolation

• Isolate VAE streaming decode state per generation session.
• Prevent decode state from leaking between concurrent or sequential sessions.
• Add transactional poison and release behavior for decode failures.
• Release VAE state as part of generation-session cleanup.

Model and parallel execution

• Update the LingBot DiT, denoising pipeline, attention configuration, and runtime integration.
• Support H100 Sage Attention and Ulysses sequence parallelism.
• Enable FSDP for the four-GPU streaming configuration.
• Align camera control, intrinsics, sampler settings, and prompt defaults with the reference H100 workflow.

Examples and interactive streaming

• Add the four-GPU H100 offline example and associated input assets.
• Update the four-GPU stream service example.
• Add bidirectional WebRTC camera control.
• Support translation through WASD/arrow keys and rotation through IJKL.
• Convert keyboard input into accumulated camera poses.
• Hold the latest generated frame while the user is idle.
• Support the tested 20-second, 321-frame streaming configuration.
• Add local, VS Code Remote SSH, and standard OpenSSH browser workflows.
• Document TURN-over-TCP configuration and coturn deployment.

Regression coverage

• Add model and attention regression tests.
• Add parallelism and runtime-baseline coverage.
• Add action-loop and camera-control tests.
• Add generation-session and cache-lifecycle tests.
• Add worker shutdown and failure-cleanup tests.
• Add VAE decode-state isolation coverage.
• Add offline and streaming example configuration tests.

[Copilot]

Pull request overview

This PR refactors the LingBot-World-Fast pipeline to use externally owned, action-driven generation sessions with explicit chunk transactions, while moving worker-local state (KV caches, scheduler/timestep selection, RNG state) into cache registries addressed by handles. It also extends the stream server lifecycle to pass a gpu_num argument into stream pipeline factories, and adds substantial unit/regression coverage plus updated examples/docs for multi-GPU streaming.

Changes:

• Introduce externally owned generation sessions + single-chunk API with transactional poison/release semantics; move persistent denoise state into worker-local cache registries keyed by cache handles.
• Add --gpu-num plumbed through stream-serve → service container → StreamPipelineService → get_service(gpu_num=...) (while keeping factories without gpu_num supported).
• Add LingBot multi-GPU offline + streaming examples, WebRTC demo/UI updates, and broad unit tests for lifecycle, control alignment, caching, and attention backends.

Reviewed changes

Copilot reviewed 36 out of 40 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/worker/test_parallel_worker_lifecycle.py	Adds lifecycle/timeout/failure/close idempotency coverage for ParallelWorker.
tests/unit/service/test_service_routes.py	Extends CLI/service tests to validate --gpu-num plumbing into stream server/service startup.
tests/unit/pipelines/lingbot_world_fast/test_stream_example.py	Verifies stream example defaults and gpu_num wiring into service/pipeline.
tests/unit/pipelines/lingbot_world_fast/test_session_cache.py	Tests cache-handle registry semantics and session-scoped VAE decode state.
tests/unit/pipelines/lingbot_world_fast/test_service_action_loop.py	Adds tests for action-driven session loop, directional controls, and session lifecycle.
tests/unit/pipelines/lingbot_world_fast/test_runtime_baseline.py	Adds runtime/session baseline tests (chunking, reproducibility, cleanup on failure).
tests/unit/pipelines/lingbot_world_fast/test_pipeline_call.py	Validates single-chunk API transaction semantics, poisoning, release, and concurrency rejection.
tests/unit/pipelines/lingbot_world_fast/test_parallelism.py	Tests Ulysses + optional FSDP wiring in denoising stage.
tests/unit/pipelines/lingbot_world_fast/test_control_alignment.py	Migrates/expands control alignment tests to new control builder + truncation APIs.
tests/unit/ops/test_attention_backends.py	Adds coverage that Sage attention import prefers TeleFuser kernel module.
tests/unit/models/test_lingbot_world_fast_dit.py	Adds coverage for unified attention usage + config propagation in LingBot DiT blocks.
telefuser/worker/parallel_worker.py	Adds failure/closed state, idempotent close, and robust wait/terminate semantics.
telefuser/service/main.py	Plumbs gpu_num into stream server initialization.
telefuser/service/core/stream_pipeline_service.py	Adds gpu_num support by introspecting get_service factory signature.
telefuser/service/core/container.py	Threads gpu_num through container stream initialization.
telefuser/pipelines/lingbot_world_fast/session.py	Introduces session/chunk request/result/status models and session-owned decode state.
telefuser/pipelines/lingbot_world_fast/service.py	Refactors streaming service loop to external session + action-driven chunk execution; adds duration limits and control integration changes.
telefuser/pipelines/lingbot_world_fast/pipeline.py	Implements externally owned session lifecycle, cache-handle init/release, and denoise via worker cache registry.
telefuser/pipelines/lingbot_world_fast/denoising.py	Converts denoising into a BaseStage with cache registry and initialize_cache/denoise_and_update_cache APIs.
telefuser/pipelines/lingbot_world_fast/control.py	Refactors control building to explicit context/builder + deferred materialization and truncation helpers.
telefuser/pipelines/lingbot_world_fast/init.py	Updates public exports to new session and chunk request/result types.
telefuser/ops/attention/backends.py	Adjusts Sage attention import priority order.
telefuser/models/wan_video_vae.py	Adds session-owned streaming decode state for incremental VAE decoding.
telefuser/models/lingbot_world_fast_dit.py	Switches to unified attention backend + adds Ulysses sequence-parallel hooks.
telefuser/entrypoints/cli/main.py	Adds --gpu-num/-g CLI option and passes it to run_stream_server.
telefuser/core/base_pipeline.py	Adds opt-out flag to skip post-call GPU memory clearing for stateful pipelines.
README.md	Updates LingBot WebRTC demo quickstart and networking/TURN guidance.
examples/stream_server/webrtc_bidirectional_demo.py	Updates UI/controls, duration-to-frame mapping, defaults, and uses threaded HTTP server.
examples/lingbot/stream_lingbot_world_fast.py	Adds a get_service(gpu_num=...) factory and multi-GPU pipeline configuration.
examples/lingbot/README.md	Adds detailed LingBot offline + streaming docs, tested limits, and troubleshooting.
examples/lingbot/lingbot_world_fast_image_to_video_h100.py	Adds offline 1/4-GPU H100 example using new session/chunk API and control source.
examples/example_config.yaml	Adds LingBot offline example entry.
docs/zh/stream_server.md	Documents --gpu-num and expands LingBot streaming/TURN guidance (ZH).
docs/zh/service.md	Updates service doc snippet referencing LingBot streaming configuration (ZH).
docs/en/stream_server.md	Documents --gpu-num and expands LingBot streaming/TURN guidance (EN).
docs/en/service.md	Updates service doc snippet referencing LingBot streaming configuration (EN).
.gitignore	Adjusts ignore rules to include LingBot example assets in-repo.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.