·
823 commits
to main
since this release
Immutable
release. Only release title and notes can be modified.
Note
If you are installing from a release, please refer to the README, INSTALLATION instructions, and any other documentation packaged with the release, not on the main branch. There may be differences between the latest main and the previous release.
The changelog will now follow, showing the changes from last release.
LLK (low-level kernels)
- Make Unpacker Strides Operation-Restorable PR 45127
- Use float literals for 0.8373f PR 48768
- Pack should not require and Uninit PR 48468
- Update ttsim version to 1.9.3 PR 48859
- BH unpack-reduce: set SrcB Unp_LF8_4b_exp for Fp8_e4m3 scaler (F3) PR 48367
- Feat: Add support for uint32 binary remainder PR 48697
- BH SFPU teardown: drop the spurious addr-mod-base reset (S2) PR 48359
- BH math config writes: stall on SFPU when writing SFPU-read PR 48360
- feat: optimize llk test build process by tracking built variant IDss PR 48890
- fast quasar compile with runtime() deduplication PR 48611
- Informational cleanups: docstrings, comments, dead code, asserts PR 48298
- ci: move llk quasar build from xlarge to large runner PR 48909
- Performance/accuracy: tanh (fp32) PR 48299
- ci: use
--dist=workstealin llk tests PR 48941 - Fix BH tilize UNPACR_NOP zerosrc encoding PR 48969
- Revert "feat: optimize llk test build process by tracking built variant IDs" PR 49020
- SFPU accuracy CSV harness PR 46361
- feat: add perf optimization audit rule set PR 49146
- Drop duplicate tensix_sync() in set_dest_fmt(uint32_t,int) PR 49189
- Write ZEROACC-mode cfg after the drain in llk_math_hw_configure (defensive) PR 49186
- Add quasar support to clangd setup script PR 49225
- WH SFPU medium complexity kernels cleanup PR 48696
- Some more SFPI deprecations avoidance PR 49239
- Document the to_from_int8 reconfig caller contract for stale unsigned/INT8 bits (F5) PR 48363
- fix: optimize softplus perf for bfp16 PR 49073
- TTI -> sfpi conversions gap PR 49362
- Invalidate src zero-flag tracker after WH UInt16 B2D restore (M1) PR 48365
- Cleanup: Remove deprecated face-geometry compute and LLK APIs from PR #44281 PR 48915
- Remove erroneous writes of DBG_ARRAY_RD_CMD=0 PR 48647
- Use new sfpi::clamp API PR 49408
- Add multi-file support to LLK run_test.sh PR 49349
- chore: update ttsim version to 1.9.4 PR 49461
- Remove copies in operation_init/operation_check PR 49237
- Store SFPU accuracy results as Parquet PR 49084
- Repro and quick fix 44750 PR 49085
- fix: replace tt_sfpconfig with sfpi assignment PR 49387
- Fix Quasar unpack transpose reconfig escape PR 49338
- Do not use deprecated sfpi::vConst0 PR 49522
- Quasar: Tiny tile port for Reduce LLK PR 42809
- Mask SRC_ADDR_OFFSET write in set_packer_dest_registers (tt-llk#1659 N1) PR 49192
- Dedup BFP exp-section-size formula into bfp_exp_section_size() helper PR 48177
- Unsigned Integer Support for relu-related ops PR 49373
- move_d2a_row_broadcast_fixed_face: add SRCA_VLD stall before MOVD2A run (F-WH-13/F-BH-12, tt-llk#1664) PR 49211
- WH SFPU complex llks cleanup PR 48925
- Remove dead unpacker writes of INT8 / SrcB word-1 config bits PR 49637
- Add /perturb skill: timing-perturbation reproducer for flaky kernel races PR 48983
- race-audit: don't conclude from a gap — ground negatives (missing ISA doc) + complete the cfg-write mechanism sweep PR 49006
- Use the TT_METAL SFPLOADMACRO disable env var in LLK ttsim regression PR 49638
- Route LOADMACRO-disabled SFPU MAX/MIN reductions through manual paths PR 49634
- chore: update ttsim-version PR 49664
- Fix operand A unpack_tilizeA_B l1 index PR 49485
- Build SFPU binary-bcast lane masks from LTILEID instead of SFPCONFIG PR 49682
- shift and signbiit -> sfpi PR 49471
- Add pack and unpack reconfig tests PR 48724
- Document SDPA SFPU reduce Dst scratch lifetime PR 49687
- fix: eliminate OOM error by moving build artifacts from tmpfs to disk PR 49794
- Drop stochastic-rounding variants from LLK unpack regression sweeps PR 49669
- Add 32bit dest TilizeA_B test PR 49579
- initial quasar support PR 49105
- TensorShape support and verification for llk_unpack_A PR 47045
- fix: prefix
_sfpu_load_config32_withckernel::mathPR 49896 - Program BH FP8 E4M3 packer input format as Float16 PR 49909
- Blackhole SFPU SIMPLE-bucket sfpi conversions PR 49820
- fast reduce unpack for 32x32 tiles PR 49715
- chore: update ttsim version to 1.9.6 PR 49834
- Quasar: derive addrmod thread id from COMPILE_FOR_TRISC PR 49790
- LLK: fix llk_unpack_tilize_block cross-tile-row stride for tiny tiles PR 49635
- Init Quasar LLK perf test support PR 49348
- Enable true INT32 for relu_min/relu_max/relu6 PR 49721
- Add
--opflag to filter tests by SFPU op PR 49832 - Blackhole SFPU MEDIUM-bucket sfpi conversions (net wins) PR 49826
- LLK] perf: add cross-thread zone-alignment barrier (sync_point) to the profiler PR 49741
- 16x32 tiny-tile support for SDPA reduce_block_max_row + sub_bcast_col_custom PR 49270
- Fix race condition on dest access register rmw PR 50123
- feat: add support for add and sub binary sfpu kernels on Quasar PR 50138
- Close gap on sfpu functional test coverage PR 49782
- close SFPU functional tests gap part 2 PR 50213
Metalium (tt-metal core)
- SFPI 7.65.0 737 PR 48789
- #48805: Materialize __tdata_lma address once in TRISC kernel startup (trisck.cc) PR 48806
- Feature: Add optional num_bytes parameter to D2HSocket::has_data() PR 48626
- Cleanup : Move disaggregation from experimental to internal PR 48638
- #48793: adjust firmware timeout for functional sim PR 48803
- Cleanup: Move internal mesh CQ methods from public to internal header PR 44291
- Avoid enumeration of disabled dfb implicit sync PR 48740
- Enabling DFBs in Quasar fast-dispatch flow PR 48840
- Kernel scratchpad for Metal 2.0 PR 48554
- emule: multichip fiber engine + fabric/CCL teleport (8-chip loudbox) PR 48548
- Fixed fabric tests across all workflows. PR 45056
- Bump ttsim version to v1.9.3 PR 48855
- fix(test): gate tt-emule ASAN sanitizer tests to the emule build PR 48862
- perf(dispatch): reserve per-core RTA vectors per kernel group PR 48034
- 38935: Migrate tt::stl namespace to ttsl PR 48875
- Feat: Add support for uint32 binary remainder PR 48697
- refactor: use std algorithms in host/test code PR 48041
- Automated UMD Bump 03.07.2026 PR 48901
- Fix from_flatbuffer(SubDeviceId vector) iterating over the empty vector PR 48181
- Subtract dispatch links from get_num_links PR 45352
- Compute: make Quasar-unsupported APIs fail at compile time PR 48937
- #48918: Remove temporarily added test by emule asan PR PR 49116
- noc_async_read/write_shard: NOC id silently binds to offset parameter (#48263) PR 49054
- Feature: increase max number of RTAs beyond 341 via WRITE_PACKED_LARGE_UNICAST PR 48686
- Tolerate concurrent temp-file renames in kernel dir scan PR 48752
- WorkerConfigBufferMgr::PrintStatus: dead loop condition skips queued entries (#48268) PR 49057
- Align LocalTensorAccessor device API to its sister Scratchpad API PR 48867
- Fix real-time profiler not recording the last program PR 48844
- clang-tidy: stub linker to skip link steps during analysis PR 48412
- Fix DeviceCommandCalculator/DeviceCommand size mismatches in dispatch PR 46564
- Add Dram deployment tests PR 49128
- emule: restore Object-Intent ASAN under the fiber engine PR 48965
- Add CODEOWNERS for emule asan PR 49165
- API for offline kernel compile PR 43851
- Fix a linking bug and silently skipping all Torus groups PR 48902
- D2HStreamService: metadata-only mode PR 48865
- Enable trace capture and replay on Quasar PR 49172
- Fix deprecated TTNN/Metalium API usage surfaced by tt-train clang-tidy PR 49088
- Add total num entries getter on DFB PR 49197
- Add 193 Metal 2.0 DFB tests on Quasar PR 45159
- feat: add asserts for unsupported features of compute API on Quasar PR 49287
- Incorrect watcher check & usage around Quasar SRAM uncached memory addresses PR 48624
- fix tensor layout validation for conv2d PR 47414
- SFPI 7.66.0 744 PR 49363
- emule: translate raw persistent-L1 reinterpret_casts in JIT kernels PR 49319
- emule runner: support downstream kernels with configured include paths + namespaced CT/RT args PR 49321
- Feature: multi-config layer support PR 48620
- Cleanup: Remove deprecated face-geometry compute and LLK APIs from PR #44281 PR 48915
- Add 4x8wh fabric validation config PR 49306
- Remove erroneous writes of DBG_ARRAY_RD_CMD=0 PR 48647
- remove
TensorLayout::with_memory_configPR 48720 - ci: add pytest-repeat to dev requirements PR 49406
- Fix inconsistent noc usage for fabric workers PR 49396
- Streaming weights mode for the tensor prefetcher recv-contig path PR 47773
- Bug fix: chunk large-unicast RTAs by prefetch command size (fixes on_eth hang, #49336) PR 49409
- DS Prefill: Runner mock integration PR 49234
- Bump ttsim version to v1.9.4 PR 49444
- Add NOC-distinctness legality check to the Metal 2.0 API PR 49446
- Feature: Add posted write support to send_payload_*_from_address fns PR 49449
- Add Ethernet Depolyment Tests PR 49215
- fix: replace tt_sfpconfig with sfpi assignment PR 49387
- Avoid FD Mesh CQ teardown race PR 49386
- Profiler L1 buffer accumulate mode PR 48506
- Adding FD support to catch watcher test mode errors PR 48842
- Quasar: read NoC cmdbuf registers via RoCC PR 46234
- Expand runtime CI to multichip topologies (#46302) PR 46694
- Quasar: Tiny tile port for Reduce LLK PR 42809
- Fix watcher Tile Counter test hang PR 49450
- Fix std::terminate/use-after-free on watcher thread join ti… PR 49419
- Unblock Blaze kernel compilation with mock device PR 49440
- Avoid accesses to local cb/dfb interface on MATH PR 49503
- Bump jupyterlab from 4.5.7 to 4.5.9 in /tt_metal/python_env PR 48577
- Enable
dispatch_son Quasar fast dispatch PR 49541 - emule: fix fiber-scheduler teardown hang from non-atomic W_/generation_ publish PR 49497
- Update WH/BH regression suites to expand ASan testing PR 49525
- Unsigned Integer Support for relu-related ops PR 49373
- Automated UMD Bump 10.07.2026 PR 49572
- Recategorize DFB tests and remove Quasar duplicates PR 49500
- Scale DRAM-backed
prefetch_q_entriesby number of command queues PR 49604 - Bump ttsim version to v1.9.5 PR 49639
- emule: exclude Emule from mock-only SD compile + fix ASAN symbolizer debuginfod hang PR 49609
- Fabric Mux V2 PR 48675
- Enable NoC sanitize in FD on QSR PR 49613
- Compute shard ID in the tensor accessor using shard coord PR 49594
- Fix ODR violation in DFB tests PR 49690
- Refactor Metal 2.0 hardware configs PR 49516
- Dispatch telemetry SMC Control PR 48762
- binary_ng: fix in-place add descriptor rebuild-on-cache-hit (#48928) PR 49248
- Add debug-verbosity Tracy zones PR 46386
- Fix fabric fatal on inter-mesh (Z) links PR 49769
- Fix: Make NumaAwareExecutor::shutdown_ atomic to avoid a data race PR 49558
- Add fabric APIs for eager-compile model + control-plane bindings PR 48851
- Remove MeshTensor backdoor for ttnn PR 49777
- Blitz superpod mapping determinism testing PR 46213
- Add socket send/recv + mesh-socket rank-addressed support + protobuf import binding PR 48852
- Align debug_pause_msg_t on QSR PR 49767
- Make read_tile_value templated and add test for read_tile_value/get_tile_address PR 49617
- Bug fix: Clear stale remote CB configurations PR 49704
- Automated UMD Bump 11.07.2026 PR 49643
- Bump ttsim version to v1.9.6 PR 49807
- emule: deliver fused scatter-write + atomic-inc in __emule_fabric_deliver PR 49689
- Report raw L1 overflow address on QSR watcher PR 49622
- Fix NOC_0 collision in ProgramSpecHWTest for Metal 2.0 PR 49831
- Add 32bit dest TilizeA_B test PR 49579
- Misc. updates to DM test suite PR 49642
- Swap RTA addressing key in
ProgramRunArgsPR 49432 - Cleanup: Fix CMake GLOB_RECURSE to filter by extension PR 44791
- Subtorus Pipeline Builder Support PR 48854
- Add apis to get total dfb fifo size and local views that give size/num entries for active tile counter PR 49652
- Bug Fix: Skip cross-host eth cores in deployment eth tests PR 49750
- per_token_cast_to_fp8: Add power-of-two scaling PR 49811
- Support >4GB Total L1 Size in Emulated Device PR 49821
- Adding vision model to qwen3.6 PR 49531
- Add bh-ring-stress fabric CPU-only group + regolden after tt-cluster-descriptors uplift PR 47847
- SFPI 7.67.0 761 PR 49914
- LLK: fix llk_unpack_tilize_block cross-tile-row stride for tiny tiles PR 49635
- Enable true INT32 for relu_min/relu_max/relu6 PR 49721
- disable device profiler for mock device PR 49596
- 16x32 tiny-tile support for SDPA reduce_block_max_row + sub_bcast_col_custom PR 49270
- Quickfix remove problematic warning message from telemetry PR 50133
- Pimpl GlobalSemaphore in tt-metalium API PR 49776
- Relax Quasar NoC transfer alignment PR 49955
- Metal 2.0 compute config cleanup PR 49521
- Make RowMajorPageConfig custom tile a hard error PR 50166
- Skipping read_tile_value DFB test until #50135 is resolved PR 50148
- feat: add support for add and sub binary sfpu kernels on Quasar PR 50138
- fix(fabric): don't instantiate MetalContext in PhysicalSystemDiscovery (read mock env var directly) PR 50168
TT-NN
- Mark mux-region MoE core placement tests Blackhole-only PR 48730
- Migrate MoE / a2a CCL ops to Device 2.0 API PR 47547
- Fix FP32 fuse-add path (FPU add to SFPU add) in pre-allgather Welford layernorm PR 47791
- Cleanup : Move disaggregation from experimental to internal PR 48638
- D2H service optimizations PR 48739
- Fixed fabric tests across all workflows. PR 45056
- #48660: comment out method outputs matmul warning PR 48848
- Test only: set Welford group norm DRAM Frobenius threshold to 0.01 PR 46686
- 38935: Migrate tt::stl namespace to ttsl PR 48875
- slab-aware (block-cyclic) K addressing, sparse_sdpa-matched interface PR 48772
- Feat: Add support for uint32 binary remainder PR 48697
- Fix compiler and runtime warnings in SDXL op unit tests PR 47702
- test_cbrt: make coverage RNG-version-independent (exhaustive bf16 vs fp64 golden) PR 48769
- retire data_movement legacy helpers + tt_memmove body refactor PR 47454
- #41665: Exhaustive testing Typecast op for UINT8 -> BFLOAT16 conversion PR 47926
- Fix depthwise conv1d multi height block (in0_num_blocks_h > 1) PR 48632
- mid-slab-boundary straddle PR 48808
- Tilize - float32 precision tests PR 48251
- Fix sum HW reduction PR 48578
- Untilize: enable fp32_dest_acc_en for INT32 datatype PR 47919
- perf(ccl): return slice vector by move, not element-wise copy PR 48033
- Subtract dispatch links from get_num_links PR 45352
- ttnn/argmax (multicore): remove timing-dependent k==0 done_sem reset PR 48534
- Fix unused variables in vllm test PR 48814
- Performance/accuracy: tanh (fp32) PR 48299
- Fix to_layout without an explicit memory_config PR 48936
- Fix writer kernels sourcing NoC write from get_write_ptr() instead of get_read_ptr() PR 48994
- #33882: sharding support for index fill PR 48423
- Relax DiT RMSNorm fused RMSE thresholds for bf16 low-precision RMSNorm PR 48964
- Handle single-core gather-in0 matmul CB sizing PR 48966
- Update group_norm DRAM grid validation test setup PR 48971
- Use row-wise work splitting for fast_reduce_nc sub-core grids PR 49001
- Test Only: skip fused tests failing with LLK asserts PR 49007
- tilize op sharded path optimization PR 48761
- fix causal for block-cyclic chunk_start on a non-zero boundary chip PR 48935
- Test Only: Fix fused fp32 mismatch expected error PR 49040
- Add failure-path diagnostic logging for flaky TestAsyncRuntimeAllocatedBuffers (#43725) PR 49011
- Use float literals 0.0f, 1.0f, -1.0f 2/2 PR 48784
- Bug fix: disable submesh program cache in AsyncExecutionWorksCQ0 CCL test (#46982) PR 49161
- Fix real-time profiler not recording the last program PR 48844
- Reject unaligned row-major shard widths instead of returning corrupt data PR 49122
- Migrate normalization + reduction ops to Device 2.0 API PR 49110
- Migrate shared writer unary stick kernel file to Device 2.0 PR 46748
- D2HStreamService: metadata-only mode PR 48865
- Extended ttnn.prod to support float32 PR 47409
- Adding DeepSeek-V4 MoE gate: sqrtsoftplus routed gate + new hash gate op PR 48846
- Add token-level causal mask to MSA PR 48700
- Fix non-tile-aligned sharded LayerNorm and RMSNorm PR 48856
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49138
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49132
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49134
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49135
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49137
- Fix deprecated TTNN/Metalium API usage surfaced by tt-train clang-tidy PR 49088
- unified_routed_expert_ffn: add SwiGLU-OAI activati… PR 47825
- Fix GIL deadlock PR 49202
- Migrate all-gather + send/recv CCL ops to Device 2.0 API PR 47537
- DeepSeek V3 prefill: bring up Ring-8 and Ring-4 on a single-galaxy sub-torus PR 48225
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49133
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49136
- Migrate data_movement roll/concat/scatter/fill_pad to Device 2.0 API PR 48834
- Fix #48928: descriptor rebuild-on-cache-hit (ResNet50 non-trace, ~12x) PR 49159
- Feature: ttnn.outer through broadcast multiply PR 44484
- #31024: ttnn.pad_to_tile_shape API deprecation PR 48702
- LayerNorm: Added FP32 support for LayerNormPostAllGatherAdd and validation of welford & legacy configuration for FP32 PR 48226
- Add unsigned integer support for relu and reglu PR 49212
- Universal input/output support for ttnn.pad PR 47507
- Fix PCC assertion failures for single-element bfloat8_b tensors PR 49267
- #48688: [ttnn][tilize] Fix crash when DRAM-backed sharded tensor routed into zero-copy optimized factory PR 49213
- Fix ttnn.outer TT_FATAL on row-major bf16 vectors (matmul dispatch not tilized) PR 49332
- prod: remove stale cb_in1_obj reference breaking prod_nc JIT compile PR 49235
- fix tensor layout validation for conv2d PR 47414
- Feature: multi-config layer support PR 48620
- Bug Fix: fall back to generic tilize for DRAM sharded inputs PR 49261
- remove
TensorLayout::with_memory_configPR 48720 - conv1d: comment out bf8/bf8 in_channels=3 case (workaround for #49393) PR 49394
- Fix CCL get_num_links collapsing to 1 on Ring/Torus wrap-around PR 45218
- Bug Fix: Resolve failing TTNN comparison test PR 49188
- Undoing layernorm test skips PR 49416
- Streaming weights mode for the tensor prefetcher recv-contig path PR 47773
- ttnn/eltwise/unary: Buffer* rt-arg bindings to skip rebuild-on-cache-hit (#48928) PR 49301
- DS Prefill: Runner mock integration PR 49234
- ttnn: give ds-prefill its own CMake subdirectory aggregator (reference PR) PR 49441
- Bound sparse MLA indexer score + top-k to written prefix (kv_len) PR 49359
- fix: replace tt_sfpconfig with sfpi assignment PR 49387
- Fix test_outer BFLOAT8_B scalar assertion PR 49455
- #30282: fix(creation): ones/zeros/full_like dtype override ignored for TILE-layout device tensors PR 49209
- Fix namespace collision in functions.hpp in Unity builds PR 49427
- Support unaligned reshard op PR 49311
- ttnn/move: get_dynamic for sharded move — skip rebuild-on-cache-hit (#48928) PR 49289
- ttnn: shared-ownership CMakeLists.txt for all of experimental/'s movable ops PR 49454
- ttnn: eltwise + shared operations/ ownership for convolutions/ops-data-movement/mmfusedreduce PR 49448
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49131
- ds_prefill - Remove extract OP from routed expert PR 49343
- Support untilize (to_layout) from batched HEIGHT_SHARDED TILE input to interleaved row-major with non-tile-aligned H/W PR 49489
- Support same-device sender_coord == receiver_coord in PointToPoint PR 49168
- ttnn.reshape: support bfloat4_b via block-float typecast round-trip PR 49171
- Migrate Moreh Kernels from CircularBuffer to DataflowBuffer PR 49430
- Migrate eltwise Kernels from CircularBuffer to DataflowBuffer PR 49410
- #42882: Bound RM slice CB size via sub-row NOC chunking PR 49067
- #48340: Fix L1 CB clash and preserve sharding in ttnn::split default memory config PR 49044
- Fix logical_and test to compare golden against device result PR 49505
- indexer_score_msa: expose cache_batch_idx + kv_len (paged/prefill KV cache) PR 49486
- Unsigned Integer Support for relu-related ops PR 49373
- fix(ring-joint-sdpa): patch scalar runtime args on every core on cache hit PR 49511
- #49481 #49112: GLM-5.1 sparse (DSA) prefill serving through the prefill runner PR 49483
- Use realtime profiler for SDPA perf checks PR 48199
- Perf: Optimize semaphore handling in RS PR 49512
- Use Metal2 data movement role hints for binary_ng sharded kernels PR 49584
- Program-cache-hit Fast Path with In-Place Ops PR 49573
- Add support for fused swiglu to minimal matmul variants PR 48742
- Optimize deg2rad/rad2deg to use native unary scalar-multiply (~2.16x faster) PR 49663
- Fix Qwen3-32B Galaxy Top-1 0% regression: SDPA tilize_q programs SrcA for the QK-matmul operand PR 49583
- Migrate experimental SSM / reduction / CNN ops to Device 2.0 API PR 49334
- Compute shard ID in the tensor accessor using shard coord PR 49594
- Refactor matmul validation and clean up error messages PR 49360
- Enable unity (jumbo) build PR 49681
- Remove TT_METAL_SIMULATOR checks from tests PR 49633
- Refactor Metal 2.0 hardware configs PR 49516
- sparse_sdpa: use compile-time args for block-cyclic remap PR 49698
- ttnn: move all nanobind sources out of ttnn/sources.cmake PR 49543
- Migrate experimental/reduction/deepseek_moe_fast_reduce_nc to Device 2.0 API PR 49335
- Functional SFPU implementation for mean PR 48896
- Remove smuggled buffer-address runtime args (BufferBindings + get_dynamic_runtime_args) PR 49546
- Support L1–DRAM resharding and uint8 memory-config copies PR 49590
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49547
- Remove smuggled buffer-address RTAs from quasar matmul PR 49597
- nlp_create_qkv_heads(+boltz): remove smuggled buffer-address runtime args PR 49548
- binary_ng: fix in-place add descriptor rebuild-on-cache-hit (#48928) PR 49248
- Migrate embedding_backward to Device 2.0 API PR 48920
- Migrate data-movement + example ops to Device 2.0 API PR 49316
- Add fabric APIs for eager-compile model + control-plane bindings PR 48851
- Add socket send/recv + mesh-socket rank-addressed support + protobuf import binding PR 48852
- Fix test_atan2_a test failure on BH P150 PR 49728
- #19475: [ttnn] Add regression tests for untilize_with_unpadding narrow height-sharded CB page size PR 49043
- Remove smuggled buffer-address runtime args (BufferBindings + get_dynamic_runtime_args) PR 49545
- Fused distributed Layernorm + RMSNorm op for DiT models PR 48719
- Improve GEMM FLOPS benchmark and matmul tuning PR 49621
- Swap RTA addressing key in
ProgramRunArgsPR 49432 - Cleanup: Fix CMake GLOB_RECURSE to filter by extension PR 44791
- Cleanup: Move ttnn_graph_report fixture body into ttnn.graph_report PR 48559
- ds_prefill - Remove unnecessary reading of weights tensor from dispatch operator PR 48141
- Remove smuggled buffer-address runtime args (BufferBindings) PR 49544
- zero_padded_kv_cache: Support row-major caches PR 49808
- Fix bf8 scalar outer nightly tests PR 49799
- ds_prefill - L2 nightly CI fix PR 49833
- Migrate experimental DeepSeek-gating / distributed-RMSNorm ops to Device 2.0 API PR 49370
- per_token_cast_to_fp8: Add power-of-two scaling PR 49811
- Capture ttnn auto-selected program config via query_op_constraints PR 49253
- exp_ring_joint_sdpa: re-apply hash-excluded global-semaphore addresses PR 49550
- copy: Support row-major FP8 tensors PR 49809
- #42927: Tiny-tile matmul tests, validation guards, BH fused-bias CB fix PR 49632
- Fix missing return in override_runtime_args out-of-bounds guard PR 49857
- Test Only: test_concat_fuse_6u one-liner fix PR 49849
- experimental-quasar binary_ng: no-broadcast DFB factory + qualification harness PR 48676
- Migrate Conv/Pool Kernels from CircularBuffer to DataflowBuffer PR 49439
- Migrate local free-function semaphores to Device 2.0 Semaphore API PR 49598
- all_to_all_combine: drop obsolete address-in-hash to restore program-cache hits PR 49876
- #36765: untilize silently corrupts padded tensors PR 48930
- Migrate Data Movement Kernels from CircularBuffer to DataflowBuffer PR 49392
- fix hangs in Wan on quad and realign ccl multihost ring geometry introduced from PR#46138 PR 49856
- #46565: Extend universal input/output support for ttnn::gather PR 47361
- Fix HEIGHT_SHARDED width front-pad offset PR 49493
- Enable true INT32 for relu_min/relu_max/relu6 PR 49721
- #49987: Skip test for alignment mismatch in index_fill op PR 49990
- SP×TP seq-sharded DSA indexer PR 49496
- pack_scaled_fp8_kv_cache: Add mixed-format row packing PR 49812
- Use f32 intermediate values in SDPA PR 48753
- matmul: use framework default program hash (decouple descriptor helper) PR 49869
- generalized_moe_gate: drop redundant custom compute_program_hash PR 49870
- Adds fused full-tile bias support to MatmulMultiCoreReuse PR 49825
- Document I/O specs and fix examples for CCL ops PR 49861
- batch_norm RunningStatistics: drop redundant custom compute_program_hash PR 49871
- Expose ccl get_usable_topology to Python PR 49838
- #50157: static cast size_t to uint32_t in hetergeneous_data_structs.hpp PR 50159
- Remove WH-only LLK workarounds PR 49859
- l2_nightly - Skip failing test PR 50019
- Pimpl GlobalSemaphore in tt-metalium API PR 49776
- Metal 2.0 compute config cleanup PR 49521
- #50165: update resnet50/quasar and experimental/quasar ops PR 50171
- Make RowMajorPageConfig custom tile a hard error PR 50166
- Migrate reduce-scatter + neighbor-pad + strided ops to Device 2.0 API PR 47555
- Remove commented lines PR 50189
- Update ttnn.linear test to match default math fidelity computation PR 49802
- feat: add support for add and sub binary sfpu kernels on Quasar PR 50138
- #48710: gelu_bw - update Program factory, add Tanh approximation kernel PR 49498
- Add groups to conv cache hash PR 48668
- Fix SDPA and indexer JIT deprecation warnings PR 50212
- Migrate precomposed NoC addresses to Device 2.0 UnicastEndpoint PR 49567
- Qwen3.6-27B optimizations and batch support PR 50674
tt-train
- 38935: Migrate tt::stl namespace to ttsl PR 48875
- refactor: use std algorithms in host/test code PR 48041
- Add TPS, github event, and elapsed time metrics PR 47965
- Split TT-Train L2 perf tests into per-model matrix jobs and add more tests PR 47830
- Enable fused q_rope, qkv_assemble, kv_down_split in mla PR 48210
- Fix FP16 subnormal decode in safetensors loader PR 48996
- Add new training models to Galaxy perf workflow PR 44512
- Bug Fix for GRPO Device Seeding and Prompt PR 48829
- Fix rmsnorm_bw L1 OOM on large hidden dims; default Qwen3 to fused RMSNorm PR 49389
- Fix MFU calculation PR 50114
Models
- Make Unpacker Strides Operation-Restorable PR 45127
- log CHUNK_START with chunk metadata before the forward PR 48791
- Speed up VADv2 detection head PR 48213
- Fix MoEGate module PR 48532
- Add foundation reference and QB2 bring-up harness PR 48069
- GLM-5.1 sparse MLA at tp=4 (head→sequence reshard) PR 48727
- Add GEMMA4_MAX_TOKENS_ALL_USERS override for KV-cache pool size PR 48819
- Qwen2 7b model bringup to TTTv2 PR 45524
- Qwen2.5-72B bringup to TTTv2 PR 47354
- D2H service optimizations PR 48739
- Fix prefill hang and CI issues PR 48522
- fix(bge_m3): handle transformers 5.x bare-Tensor layer return in PCC test PR 48827
- mla: use sparse_sdpa in-kernel block-cyclic remap (drop host KV reorder) PR 48733
- test_cbrt: make coverage RNG-version-independent (exhaustive bf16 vs fp64 golden) PR 48769
- Fix #48796 - Add missing sub_core_grids exposed by test PR 48864
- DeepSeek-R1 distilled Qwen 14B bringup TTTv2 PR 45431
- Qwen2.5 Coder 32B bringup to TTTv2 PR 45428
- lower BERTScore mean-F1 gate 0.70 -> 0.69 PR 48908
- Qwen2.5-VL-72B b32 decode: widen tolerance 0.15 -> 0.20 (host-correlated flakiness) PR 48921
- Gemma4 Speculative Decoding [Performance] PR 48835
- Restore FP32 accumulation on SDPA + MoE matmuls to fix PCC regression (#47311) PR 48748
- Pre-allocate on-device sampling buffers before trace capture PR 48900
- Fix #46879: rope_1d decode test builds prefill trans-mat at TILE_SIZE PR 48933
- Qwen3.6-27B: tune tier-1 unit + e2e timeouts, refresh TTFT@128 target PR 48911
- #48191: Remove bert11 Large from Model perf pipeline PR 49158
- Use float literals 0.0f, 1.0f, -1.0f 2/2 PR 48784
- D2HStreamService: metadata-only mode PR 48865
- Adding DeepSeek-V4 MoE gate: sqrtsoftplus routed gate + new hash gate op PR 48846
- Fix config/tokenizer resolution on read-only HF cache mounts PR 48866
- block-cyclic per-user lightning-indexer key cache (GLM/DS sparse chunked prefill) PR 48938
- mla: sparse_sdpa multi-user read via cache_batch_idx (drop host slot-slice) PR 48888
- Trim (Single-card) Model perf pipeline to stable_diffusion only (#49179) PR 49228
- GLM-5.1 prefill integration PR 48786
- Fix non-tile-aligned sharded LayerNorm and RMSNorm PR 48856
- Add cold/warm/long scenarios to MLA perf test PR 49100
- unified_routed_expert_ffn: add SwiGLU-OAI activati… PR 47825
- Fix Llama-3.3-70B Galaxy demo parametrize mismatch (missing token_accuracy in seqlen-sweep case) PR 49259
- DeepSeek V3 prefill: bring up Ring-8 and Ring-4 on a single-galaxy sub-torus PR 48225
- #31024: ttnn.pad_to_tile_shape API deprecation PR 48702
- DS Prefill: Kimi K2.6 dequantization support PR 48443
- Adjusting expected device perf for block_8x4_layer3_moe_fabric2d PR 49071
- sparse_mla: fix DSA nightly — rotated test needs bf16/RM kvpe cache PR 49355
- Add chunked Kimi to CI PR 49035
- Cleanup: Remove deprecated face-geometry compute and LLK APIs from PR #44281 PR 48915
- Close T3K batch-1 decode gap (small models) PR 49284
- Fix inconsistent noc usage for fabric workers PR 49396
- Gemma4 Fix for release CI PR 49320
- Improve matmul performance PR 49280
- DS Prefill: Runner mock integration PR 49234
- Bound sparse MLA indexer score + top-k to written prefix (kv_len) PR 49359
- sparse_mla: PCC-gate the indexer key cache in chunked + rotated tests PR 49458
- Llama-8B eval-32: prefetcher single-batch + repeat-batch without prefetcher (unblock bh_quietbox_2) PR 49397
- sparse_mla: layer-stacked indexer cache + fix KVPE-prefix gather OOM PR 49469
- Sparse-MLA perf test improvements PR 49478
- Fix Tier CI scheduled-run failures PR 49487
- ds_prefill - Remove extract OP from routed expert PR 49343
- SDXL CI L2 Nightly fix PR 47548
- SDXL frequent fix PR 49341
- GLM-5.2 prefill: DSA cross-layer indexer reuse PR 49315
- #49481 #49112: GLM-5.1 sparse (DSA) prefill serving through the prefill runner PR 49483
- Add support for fused swiglu to minimal matmul variants PR 48742
- MiniMax-M3: chunked-prefill model path PR 49300
- Rename pipeline to Blaze Models Prefill tests and right-size timeouts PR 49462
- Ltx2.3 performance improvements + I2V implementation PR 48657
- BH-galaxy qwen3 attention + mlp PR 49457
- Fused distributed Layernorm + RMSNorm op for DiT models PR 48719
- remove ttnn stress pipeline PR 49835
- GDN: reshape heads in ROW_MAJOR; fuse typecast into tilize PR 49565
- feat: Optimized Implementation of Depth Anything V2 Large on Wormhole (Stages 1-4) PR 35565
- ds_prefill - Remove unnecessary reading of weights tensor from dispatch operator PR 48141
- warm masked-bucket fill-width programs without a full forward PR 49707
- DRAMZeroFill: Support row-major FP8 pages PR 49810
- Subtorus Pipeline Builder Support PR 48854
- zero_padded_kv_cache: Support row-major caches PR 49808
- perf(mla): make dense k_chunk=640 head-agnostic PR 49729
- move panoptic-deeplab and bevformer tests to tier-3 e2e pipeline PR 49731
- Migrate Conv/Pool Kernels from CircularBuffer to DataflowBuffer PR 49439
- Patchtsmixer bounty PR 35692
- DS Prefill: GLM-5.2 in prefill runner PR 49730
- Adding vision model to qwen3.6 PR 49531
- Add bh-ring-stress fabric CPU-only group + regolden after tt-cluster-descriptors uplift PR 47847
- Small accuracy improvements for some DiT models PR 49328
- fix hangs in Wan on quad and realign ccl multihost ring geometry introduced from PR#46138 PR 49856
- SP×TP seq-sharded DSA indexer PR 49496
- test(deepseek_v3_d_p): fix stale GLM52Adapter import in conftest PR 50001
- fix(mla): guard dense k_chunk=640 against DeepSeek-V3.1 L1 OOM PR 50005
- test(sparse_mla): size standalone GLM-5.2 index cache to num_full_indexer_layers PR 50003
- Add Janus Pro Conv2d patch embedding with parity test PR 47333
- Use f32 intermediate values in SDPA PR 48753
- Added
zero_padded_kv_cachein bh-e2e CI PR 49954 - Remove tile config from row-major tensors in deepseek_v3_b1 tests PR 49961
- #50137: add channel parameter to pad_and_fold_conv_filters_for_unity_stride PR 50153
- Bump Mochi VAE threshold PR 50160
- Add sparse MLA CCL performance benchmarks PR 49840
- Jameslee/bringup wan model ci tests PR 47856
- #50165: update resnet50/quasar and experimental/quasar ops PR 50171
- sparse MLA: gather DSA indexer top-k indices in TILE layout (fix TP-fabric hang) PR 50172
- MiniMax-M3 prefill KV-cache migration + shared prefill validation PR 49618
- Disable force-argmax sampling on Wormhole PR 50220
- Qwen3.6-27B optimizations and batch support PR 50674
- On-device fast argmax path for Qwen3.6-27B PR 50783
TT-STL
- 38935: Migrate tt::stl namespace to ttsl PR 48875
- Fix from_json for std::unordered_map to return unordered_map (not std::map) PR 49058
Infrastructure & CI
- GPT-OSS 120B unit [bh_quietbox_2]: bump job timeout 10 -> 15 min PR 48790
- GLM-5.1 sparse MLA at tp=4 (head→sequence reshard) PR 48727
- Remove whisper p150 perf test from release workflow PR 48836
- Fix prefill hang and CI issues PR 48522
- Fixed fabric tests across all workflows. PR 45056
- fix(ci): re-enable ttsim sanity tests on push to main PR 48860
- Tilize - float32 precision tests PR 48251
- vllm-ci: strip inline comment breaking generate-matrix JSON PR 48897
- ci: move llk quasar build from xlarge to large runner PR 48909
- Add ttnn-stress-tests.yaml to aggregate-workflow-data PR 48934
- Gemma4 Speculative Decoding [Performance] PR 48835
- Add end-to-end model tests under tests/emule/models PR 48816
- Add ttsim-skip-list back to auto approve PR 48951
- Fix ttsim sanity workflow input PR 48949
- Replace personal PAT with shared token in triage-ci workflow PR 48947
- Remove Evan from merge gate Slack pings PR 48950
- Remove passing tests from ttsim skip list PR 48948
- ci: use
--dist=workstealin llk tests PR 48941 - Remove passing tests from ttsim skip list PR 48981
- Pipeline reorg: T300 Fast test reorg PR 48963
- SFPU accuracy CSV harness PR 46361
- Restore FP32 accumulation on SDPA + MoE matmuls to fix PCC regression (#47311) PR 48748
- Qwen3.6-27B: tune tier-1 unit + e2e timeouts, refresh TTFT@128 target PR 48911
- ci multihost exabox: switch to github token authenticaiton PR 49075
- Split TT-Train L2 perf tests into per-model matrix jobs and add more tests PR 47830
- Add LLK kernels Copilot review instructions PR 47394
- ci: fix sd3.5 galaxy integration test PR 49126
- GPT-OSS-120B unit [bh_quietbox_2]: bump timeout 15 -> 22 min (host variance) PR 49153
- #48191: Remove bert11 Large from Model perf pipeline PR 49158
- clang-tidy: stub linker to skip link steps during analysis PR 48412
- ci: run qb2 sdpa tests on iommu-enabled runners PR 49157
- Cleanup: Add Rui and Almeet as owners of tt_metal/impl/context PR 49164
- Add pre-commit guard against smuggled buffer-address runtime args PR 49141
- Parametrize infra CICD data collection tests PR 49174
- Add CODEOWNERS for emule asan PR 49165
- Fix CODEOWNERS: tt_metal/tools CMakeLists.txt catch-all clobbering per-tool owners PR 49163
- Trim (Single-card) Model perf pipeline to stable_diffusion only (#49179) PR 49228
- GLM-5.1 prefill integration PR 48786
- Fix T3K E2E test for DeepSeek Prefill tests PR 49241
- Add outlogix team as owners of didt and deployment tests PR 49250
- Add new training models to Galaxy perf workflow PR 44512
- Disable [WH-T3K] Gemma4-26B-A4B vLLM nightly test (DRAM OOM) PR 49246
- Add CODEOWNERS entries for disaggregation internal/ paths PR 49312
- DS Prefill: Kimi K2.6 dequantization support PR 48443
- Add chunked Kimi to CI PR 49035
- Llama-8B eval-32: prefetcher single-batch + repeat-batch without prefetcher (unblock bh_quietbox_2) PR 49397
- Cap Gemma4 12B/26B vLLM nightly max_model_len at KV-cache capacity PR 49405
- Remove passing tests from ttsim skip list PR 49472
- Re-enable Gemma4-31B vLLM nightly (cap max_model_len + KV pool) PR 49495
- Bug fix: update stale workflow names in produce-data and auto-retry [skip ci] PR 49502
- ttnn: shared-ownership CMakeLists.txt for all of experimental/'s movable ops PR 49454
- Profiler L1 buffer accumulate mode PR 48506
- MINFRA-1081: infra: map CI job card_type to pipeline SKU via sku_config.yaml [skip ci] PR 49249
- Expand runtime CI to multichip topologies (#46302) PR 46694
- ttnn: eltwise + shared operations/ ownership for convolutions/ops-data-movement/mmfusedreduce PR 49448
- Add models Copilot review instructions PR 47371
- Add ttnn.instructions.md for AI PR review PR 47434
- Disable CCache Storage Helper debug log in CI PR 49532
- SDXL CI L2 Nightly fix PR 47548
- ttop-create-allocation: add poll loop PR 49524
- SDXL frequent fix PR 49341
- GLM-5.2 prefill: DSA cross-layer indexer reuse PR 49315
- upstream tests: blackhole galaxy upstream deployment tests PR 47851
- Use realtime profiler for SDPA perf checks PR 48199
- produce data: add LLK perf to workflow run list PR 49592
- Support comma-separated model filter in All Model Tests dispatch PR 49599
- Update ttsim CI to use org runners PR 49561
- Fabric Mux V2 PR 48675
- Run ring SDPA perf checks on the IOMMU-enabled QB2 runner PR 49666
- Rename pipeline to Blaze Models Prefill tests and right-size timeouts PR 49462
- ci(pr-gate): use treeless checkout for asan and wheel builds [Performance] PR 49708
- Remove TT_METAL_SIMULATOR checks from tests PR 49633
- Remove smuggled buffer-address RTAs from quasar matmul PR 49597
- ci: bump remaining actions/setup-python v5 → v6.2.0 (Node.js 20 deprecation) [skip ci] PR 47630
- ci: remove dead check-black no-op placeholder from all-static-checks [skip ci] PR 46997
- Add debug-verbosity Tracy zones PR 46386
- Blitz superpod mapping determinism testing PR 46213
- build(packaging): run dpkg-shlibdeps only on tt-metalium for sanitizer builds [Performance] PR 49716
- BH-galaxy qwen3 attention + mlp PR 49457
- fix: eliminate OOM error by moving build artifacts from tmpfs to disk PR 49794
- remove ttnn stress pipeline PR 49835
- Pipeline reorg: Single-card Frequent model and ttnn tests PR 49743
- ci: add copilot-pr-labeler job to pr-gate PR 49795
- Add metal-runtime.instructions.md for AI PR review PR 47428
- Revert "[skip ci] Update ttsim CI to use org runners" PR 49885
- ttsim host-load sidecar PR 49774
- Use priority runners for Merge gate PR 49908
- Subtorus Pipeline Builder Support PR 48854
- CODEOWNERS: fix ds-prefill test ownership override PR 49901
- Enable single galaxy tests for blaze prefill models on exabox PR 49566
- fix per-job owner_id on single_card_ttnn_models_frequent_tests.yaml PR 49905
- move panoptic-deeplab and bevformer tests to tier-3 e2e pipeline PR 49731
- Route LLK smoke tests to event-appropriate runners PR 49935
- experimental-quasar binary_ng: no-broadcast DFB factory + qualification harness PR 48676
- Add cse developer group as codeowner for qwen36 PR 49952
- Adding vision model to qwen3.6 PR 49531
- Add bh-ring-stress fabric CPU-only group + regolden after tt-cluster-descriptors uplift PR 47847
- ci: docker SBOM and bake optimizations PR 49882
- refactor(pr-gate): remove duplicate wheel build from the PR gate [Performance] PR 49897
- Add Janus Pro Conv2d patch embedding with parity test PR 47333
- ci: remove nightly/comprehensive sweep + validation cron schedules PR 49891
- Add minimax m3 prefill owners PR 49906
- Organize merge gate by team PR 49978
- Added
zero_padded_kv_cachein bh-e2e CI PR 49954 - Modular Docker build with LLM: deepseek-pd-worker image PR 49709
- ci: remove bh_quietbox_2_iommu (merge back in to bh_quietbox_2) PR 50127
- Reduce logging level to INFO in chunked prefill PR 50164
- Jameslee/bringup wan model ci tests PR 47856
- MINFRA-1254: Rename sp to sc in workflows PR 50088
- Qwen3.6-27B optimizations and batch support PR 50674
- ci: use ubuntu-slim runner for trivial ubuntu-latest jobs PR 49992
- docker: use COPY --link for prebuilt tool/venv layers PR 50890
- fix(ci): stop anonymous Docker Hub pulls for SBOM attestation scanner PR 51203
- Add missing VBench environment variable PR 50767
Documentation
- 38935: Migrate tt::stl namespace to ttsl PR 48875
- #40182: Update sfpi documentation PR 48788
- Use float literals 0.0f, 1.0f, -1.0f 2/2 PR 48784
- Fix deprecated TTNN/Metalium API usage surfaced by tt-train clang-tidy PR 49088
- #31024: ttnn.pad_to_tile_shape API deprecation PR 48702
- fix: replace tt_sfpconfig with sfpi assignment PR 49387
- Remove vConst{0,1,Neg1,0p8373} description PR 49413
- Add CCL performance tuning tips to tech reports PR 49595
- Add debug-verbosity Tracy zones PR 46386
- Improve GEMM FLOPS benchmark and matmul tuning PR 49621
- Document sfpi API additions PR 49915
- Qwen3.6-27B optimizations and batch support PR 50674
Tooling
- Guard against duplicate --hosts and document MPI interface validation failures PR 47938
- docs(exabox): make the validation/recover workflow agent-friendly and Confluence-sourced PR 48804
- 38935: Migrate tt::stl namespace to ttsl PR 48875
- #0 - Exabox system health check wrapper script added. PR 48593
- Add 4x8wh fabric validation config PR 49306
- recover.sh: tag tt-smi reset output with hostname PR 49367
- run_validation.sh: stream tt-smi reset log tagged with hostname PR 49385
- Add Ethernet Depolyment Tests PR 49215
- Enable ETH deployment tests in health-check diag tiers PR 49484
- Update analyze_validation_results.py PR 49411
- Profiler L1 buffer accumulate mode PR 48506
- Fix eth deployment gtest filters so the tests actually run PR 49735
- Subtorus Pipeline Builder Support PR 48854
- Update tt-exalens version to 0.3.27 PR 49801
Other
- 38935: Migrate tt::stl namespace to ttsl PR 48875
- Add end-to-end model tests under tests/emule/models PR 48816
- clang-tidy: stub linker to skip link steps during analysis PR 48412
- Add Dram deployment tests PR 49128
- Add pre-commit guard against smuggled buffer-address runtime args PR 49141
- emule: DRAM-resident 2-chip CCL microtests (all_gather + point_to_point) (#229) PR 49384
- Cleanup: Limit oneAPI repository to TBB packages in install_dependencies.sh PR 46278
- #49452: cap pytest --durations report at 25 instead of unbounded [skip ci] PR 49530
- Cleanup: Replace deprecated apt-key with gpg --dearmor in install_dependencies.sh PR 46276
- fix(ring-joint-sdpa): patch scalar runtime args on every core on cache hit PR 49511
- Use realtime profiler for SDPA perf checks PR 48199
- Fabric Mux V2 PR 48675
- Run ring SDPA perf checks on the IOMMU-enabled QB2 runner PR 49666
- Add CCL performance tuning tips to tech reports PR 49595
- Fix UnboundLocalError in mesh_device fixture (scalar device count branch) PR 46900
- Add debug-verbosity Tracy zones PR 46386
- pytest.ini: register timeout marker to suppress unknown-mark warnings in ttsim [skip ci] PR 46252
- Fix header only packages for local PR 46226
- Fix local cadical PR 46222
- Bump black from 24.3.0 to 26.3.1 in /tt_metal/python_env PR 39756
- Test Only: Fix apparent A2A Dispatch prefill test hang PR 49788
- Cleanup: Move ttnn_graph_report fixture body into ttnn.graph_report PR 48559
- exp_ring_joint_sdpa: re-apply hash-excluded global-semaphore addresses PR 49550
- all_to_all_combine: drop obsolete address-in-hash to restore program-cache hits PR 49876
- ci: remove nightly/comprehensive sweep + validation cron schedules PR 49891
- Use f32 intermediate values in SDPA PR 48753
- Fix concat_fuse mesh mapper to shard along the concat dim PR 49865
- Expose ccl get_usable_topology to Python PR 49838
- ci: remove bh_quietbox_2_iommu (merge back in to bh_quietbox_2) PR 50127