Skip to content

feat(ops): add vLLM-aligned reshape_and_cache_flash - #883

Merged
voltjia merged 1 commit into
masterfrom
feat/add-reshape-and-cache-flash
Aug 6, 2026
Merged

feat(ops): add vLLM-aligned reshape_and_cache_flash#883
voltjia merged 1 commit into
masterfrom
feat/add-reshape-and-cache-flash

Conversation

@voltjia

@voltjia voltjia commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add the public ReshapeAndCacheFlash operator and Python binding surface aligned with vLLM's public reshape_and_cache_flash wrapper.
  • Reuse the canonical stride-aware CUDA kernel from both the new operator and deprecated PagedCachingInfinilm, without changing or removing the legacy public API.
  • Cover NHD/HND cache strides, padded source strides, scalar/per-head scales, supported input dtypes, shortened slot mappings, and negative slots.

Motivation

PagedCachingInfinilm is an InfiniLM compatibility API. This PR adds a canonical serving-framework-aligned replacement so consumers can migrate without removing the deprecated interface.

No issue is linked; this is part of the ongoing InfiniLM operator migration.

Type of Change

  • feat - new feature / new operator / new platform
  • fix - bug fix
  • perf - performance improvement (no behavioral change)
  • refactor - code restructuring without behavior change
  • test - adding or fixing tests only
  • docs - documentation only
  • build / ci - build system or CI configuration
  • chore - tooling, formatting, or other non-code changes
  • Breaking change (requires a ! in the Conventional Commits prefix or a BREAKING CHANGE: footer)

Platforms Affected

  • CPU (WITH_CPU)
  • NVIDIA (WITH_NVIDIA)
  • Iluvatar (WITH_ILUVATAR)
  • MetaX (WITH_METAX)
  • Cambricon (WITH_CAMBRICON)
  • Moore (WITH_MOORE)
  • Ascend (WITH_ASCEND)
  • PyTorch C++ bindings (WITH_TORCH)
  • Build system / CMake / CI
  • Python bindings / user-facing API

API Alignment

InfiniOps API Alignment target Upstream public API InfiniOps C++ adaptation
reshape_and_cache_flash(key, value, slot_mapping, k_scale, v_scale, kv_cache_dtype, key_cache, value_cache) -> void vLLM reshape_and_cache_flash(key, value, key_cache, value_cache, slot_mapping, kv_cache_dtype, k_scale, v_scale) -> None Preserve the relative order of logical input tensors, place the string attribute after inputs, and move writable key_cache / value_cache outputs to the end, as required by CONTRIBUTING.md.

All eight material upstream parameters are represented. This PR adds one canonical overload.

Smoke Test Result

Remote environment: ssh nvidia, image accelerator-dev/nvidia:latest.

python3 -m pip install . --force-reinstall --no-build-isolation --no-deps \
  --config-settings=cmake.define.AUTO_DETECT_DEVICES=OFF \
  --config-settings=cmake.define.AUTO_DETECT_BACKENDS=OFF \
  --config-settings=cmake.define.WITH_CPU=ON \
  --config-settings=cmake.define.WITH_NVIDIA=ON \
  --config-settings=cmake.define.INFINI_RT_ROOT=/opt/infinirt \
  --config-settings=cmake.define.INFINI_OPS_SMOKE_BUILD=ON
Successfully built InfiniOps

python3 -m pytest tests -m smoke -q
91 passed, 22 skipped, 10043 deselected in 20.35s

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
NVIDIA Yes Smoke build/test passed; focused build passed Focused new/legacy tests: 54 passed, 30 CPU-only legacy skips
Iluvatar Yes Not run locally; CI required Shared generic CUDA provider
MetaX Yes Not run locally; CI required Shared generic CUDA provider
Cambricon No N/A - not affected N/A
Moore Yes Not run locally; CI required Shared generic CUDA provider
Ascend No N/A - not affected N/A
Focused test output
python3 -m pytest tests/test_reshape_and_cache_flash.py \
  tests/test_paged_caching_infinilm.py -q -rs
54 passed, 30 skipped in 5.68s

python3 -m pytest tests/test_generate_wrappers.py -q
22 passed in 2.12s

The 30 skips are the existing PagedCachingInfinilm tests on CPU, where that operator has no implementation.

Benchmark / Performance Impact

N/A. This PR adds an interface and reuses the existing cache-write kernel; it does not claim a performance change.

Notes for Reviewers

  • The new public signature follows vLLM's Python wrapper, with only the repository-required input/attribute/output grouping adaptation.
  • The implementation currently supports kv_cache_dtype == "auto". FP8/NVFP4 behavior can be added later without changing the public signature.
  • PagedCachingInfinilm remains deprecated and source-compatible. Its launcher directly includes the canonical reshape_and_cache_flash/kernel.cuh; no forwarding header or extra compatibility layer is introduced.
  • Exact formatting checks passed with clang-format==21.1.8 and ruff==0.15.22.

@voltjia
voltjia marked this pull request as ready for review August 6, 2026 07:50
@voltjia
voltjia requested a review from a team August 6, 2026 07:50
@voltjia
voltjia merged commit e733e32 into master Aug 6, 2026
18 of 20 checks passed
@voltjia
voltjia deleted the feat/add-reshape-and-cache-flash branch August 6, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant