Sync with Microsoft ONNX Runtime - 01092026 - #1276
Merged
Merged
Conversation
…t#32329) ### Description Disable output value verification in `GatedDeltaNetTest.MalformedCuSeqlensIsClamped`. The test validates that malformed device-provided offsets are clamped safely and do not fault; output values are intentionally unspecified. For decreasing offsets such as `{0, 48, 16}`, tokens 48-63 are not written. A CUDA allocator may return memory containing NaNs, and `EXPECT_NEAR` rejects NaN regardless of the configured tolerance. `verify_output=false` still runs the session and fetches all outputs while skipping the invalid element-wise comparison. ### Testing - `ninja -C build/cu130/Release onnxruntime_provider_test` - `build/cu130/Release/onnxruntime_provider_test --gtest_filter=GatedDeltaNetTest.MalformedCuSeqlensIsClamped` - `clang-format --dry-run --Werror onnxruntime/test/contrib_ops/gated_delta_net_op_test.cc`
### Description - Build a compact fpA_intB CUDA kernel set by default: FP16 activations, INT4/INT8 weights, scale-only quantization, and `block_size=32`. - Add `onnxruntime_USE_FPA_INTB_GEMM_FULL` to retain the complete BF16, zero-point, bias, larger-block-size, and native Hopper kernel matrix when required. - Identify the compact configuration with `USE_COMPACT_FPA_INTB_GEMM` and exclude native Hopper kernels from it. - Support the compact non-Hopper kernels on SM75 and newer devices, using SM75 below Ampere, SM89 on Ada, and the SM80 compatibility path otherwise. - Keep eligibility, workspace estimation, profiling, prepacked-weight validation, tests, diagnostics, and documentation aligned with the compact contract. ### Motivation and Context The default fpA_intB build compiled many kernel variants that are not required by the FP16 INT4/INT8 RC model contract. A compact default reduces the generated CUDA kernel set while preserving the full matrix behind an explicit build option. The compact kernels are non-Hopper kernels rather than SM80-only kernels: they support compute capability 7.5 and newer. Native Hopper TMA/WGMMA kernels remain restricted to SM90 and the full build. ### Testing - Compact CUDA provider and `onnxruntime_provider_test` build. - Full-mode `onnxruntime_providers_cuda_llm` build. - `MatMulNBitsWorkspace.*`: 14 tests passed. - `Fp16Int8GroupwiseTest.*` and `Fp16Int4GroupwiseTest.*`: 2 tests passed on SM90 through the non-Hopper compatibility path. - Changed-file `lintrunner`, Python syntax validation, and `git diff --check` passed.
…icrosoft#29838) ### Description ONNX operators L2norm and DeQuantizeLinear are supported with the latest MLAS and verified for test failures. Removing them from skipped list as it improves op coverage for backends as well. Co-authored-by: hdharpure9922 <harshal.dharpure@intel.com>
## Description Enables the CPU GroupQueryAttention implementation to use `attention_bias` with `sliding_window_cache`. This is needed by speculative decoding with sliding-window attention, including calls that provide explicit `position_ids` for RoPE. ## Summary of Changes ### CPU GQA | File | Change | |------|--------| | `onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc` | Derives the absolute KV origin for each cache-relative batch and forwards it to attention implementations. | | `onnxruntime/contrib_ops/cpu/bert/gqa_attention_base.h` | Applies the absolute bias-column offset in quantized, unquantized, flash, and non-flash paths, with per-batch fallback for differing origins. | | `onnxruntime/contrib_ops/cpu/bert/group_query_attention_helper.h` | Allows windowed attention bias only for callers that explicitly support absolute bias offsets; CUDA and WebGPU behavior is unchanged. | ### Tests - Adds a deterministic CPU regression covering post-eviction bias indexing with explicit `position_ids`. - Extends windowed-cache parity coverage to combine attention bias, non-default explicit position IDs, repeated eviction, and forced non-flash dispatch. ## Testing - `cmake --build build/ci_cpu/Release --target onnxruntime_provider_test -j 8` - `build/ci_cpu/Release/onnxruntime_provider_test --gtest_filter=GroupQueryAttentionTest.WindowedCacheAttentionBiasWithPositionIds_CPU` - `ORT_GQA_DISABLE_FLASH_ATTENTION=1 build/ci_cpu/Release/onnxruntime_provider_test --gtest_filter=GroupQueryAttentionTest.WindowedCacheAttentionBiasWithPositionIds_CPU` - `clang-format --dry-run --Werror onnxruntime/contrib_ops/cpu/bert/group_query_attention_helper.h onnxruntime/contrib_ops/cpu/bert/gqa_attention_base.h onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc onnxruntime/test/contrib_ops/group_query_attention_op_test.cc` - `python3 -m py_compile onnxruntime/test/python/transformers/test_gqa.py` ## Motivation and Context A windowed KV cache stores resident rows in cache-relative coordinates after eviction, while `attention_bias` remains indexed by absolute sequence position. The previous validation rejected the combination to avoid silently reading incorrect bias columns. This change carries the per-batch absolute cache origin into the CPU attention paths so resident column zero maps to the correct absolute bias column. Explicit `position_ids` remain consumed by RoPE before the cache-relative transition. ## Checklist - [x] Tests added/updated - [x] No breaking changes - [ ] Documentation updated (not applicable; no public API change)
### Description Fix build break ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->
…rosoft#32306) The 8x16x16 subgroup-matrix templates hard-code 32 lanes per subgroup and derive sg_index/sg_lane from it. Request the size explicitly with SetSubgroupSize(32).
) ### Description Adds a `TensorScatter` circular-mode regression test where the batch dimension exceeds the maximum sequence length. The case also wraps sequence writes from slot 3 to slot 0, verifying that modulo applies only to the sequence coordinate while every batch prefix remains unchanged. ### Motivation and Context onnx/onnx#8353 identified that applying modulo to every coordinate in the full index tuple can fold one batch into another. ONNX Runtime's CPU and CUDA kernels already preserve prefix coordinates, but the existing tests use prefix dimensions smaller than the maximum sequence length and cannot distinguish the incorrect behavior. This test makes the regression observable: with `batch_size=5` and `max_sequence_length=4`, a full-tuple modulo maps batch 4 to batch 0. Related: onnx/onnx#8353 ### Testing - `cmake --build build/cpu_only/Debug --target onnxruntime_provider_test --parallel` - `./onnxruntime_provider_test --gtest_filter=TensorScatterTest.Circular_BatchLargerThanMaxSequenceLength` - `./onnxruntime_provider_test --gtest_filter=TensorScatterTest.*` (23 passed) - `lintrunner onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc` Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Copilot-Session: 427bc0f4-eaf2-406e-95ad-b5b1af6ccdc6
…U variants (microsoft#32118) ### Description `OrtExtendedHandlers()` had no entries for `com.microsoft.Gelu`, `FastGelu` and `QuickGelu`, and the ONNX map was missing `Elu`, so layout propagation stopped at the activation and left a transpose wedged between the NHWC Conv and the activation where `ConvActivationFusion` never saw them adjacent. Results stayed correct, the fusion just silently did not fire. `FastGelu` needs a bespoke handler rather than a table entry because its optional bias is pinned to the last dimension, and `broadcast_node_handler` would rank-normalize it into a rank-4 bias that fails `bias_gelu_helper::CheckInputs` outright, so the handler pushes the transpose only when there is no bias. `Elu` lives here rather than in microsoft#32117 because the transpose optimizer is shared cross-EP infrastructure, not WebGPU code. This also registers `FastGeluFusion` for WebGPU, which was cpu, cuda and dml only, since `GeluFusion` and `BiasGeluFusion` already landed in microsoft#32053. Covered by 6 fusion tests and 5 transpose-optimizer tests, including a negative test that `FastGelu` with a bias is left alone. ### Motivation and Context Models often already contain contrib GELU nodes, since `QuickGeluFusion` and `GeluFusion` run before layout transformation, so without these handlers the Conv activation fusion never fires on exactly the models most likely to benefit. microsoft#32117 adds the WebGPU side of the same work and needs these handlers to fuse end to end. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Ananya Anand <t-anaanand@microsoft.com>
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
August 31, 2026 20:37
hdharpure9922
self-requested a review
September 1, 2026 04:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.