Skip to content

v0.76.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 Aug 04:10
· 1009 commits to main since this release
Immutable release. Only release title and notes can be modified.
v0.76.0
e99f0cf

Note

If you are installing from a release, please refer to the README, INSTALLATION instructions, and any other documentation packaged with the release, not on the main branch. There may be differences between the latest main and the previous release.

The changelog will now follow, showing the changes from last release.

This release was generated by the CI workflow https://github.com/tenstorrent/tt-metal/actions/runs/31454192233

AI IP

LLK (low-level kernels)

  • Fix perf_eltwise_binary_sfpu compile break (formats.math non-type template arg) PR 50281
  • Make golden FTZ threshold format-specific PR 50082
  • SFPU init: consolidate common+specific per-op, drop the hoist (#50381) PR 50399
  • Fix sfpu tests PR 50413
  • Fix eltwise binary dest-reuse for ELWMUL and HiFi partial faces. PR 50658
  • Fix broken perf tests missing CLAMP_NEGATIVE kernel arg PR 50701
  • quasar sfpu: shared foundation (config_hash + LREG11 reload) PR 50751
  • Add missing tiny-tile TensorShapes to LLK coverage tables (#50196) PR 50286
  • Move pack relu config from pack init to pack hw config PR 50849
  • Assert SFPU destination indices PR 50860
  • Add "memory" clobber to vector_load/vector_store asm PR 51046
  • Reserve 2 words per open profiler zone to prevent buffer ov… PR 50988
  • Split WH mul-reduce-scalar UNPACR_NOP zero-source and set-dvalid PR 51089
  • Bug fix: Define clamp parameter for SFPU comparison perf tests PR 51118
  • widen ring_size field for compute side dfb structs to address available l1 in quasar PR 50800
  • Add missing dest-format configs to stop Quasar tests passing by accident. PR 50728
  • Fix fidelity-phase count in direct-indexed eltwise binary PR 51115
  • Retire Blackhole per-test chip-reset/bin-pack workaround now that #45727 fixes the root cause PR 51157
  • fix: switch to exalens 0.3.29 when running llk tests PR 50002
  • Unary broadcast unpack to dest support PR 51338
  • Fix failing reduce tests - keep LoFi for Maxpool and pass in tensor_shape to reduce_tile PR 51166
  • isclose: compare Inf/NaN bit patterns as vInt, not float magnitudes PR 51537
  • Initialize reduce block max trigger semaphores in LLK test PR 51638
  • Unary broadcast: Fix ALU data format config at init and perf test dvalids PR 51676
  • fix(sfpu): correct repair-loop direction in fmod/remainder kernels PR 51663
  • Fix LLK perf report schema PR 51949
  • LLK perf tests: raise loop factors above 1k cycles, add LOOP_FACTOR where missing PR 51912
  • Unify sfpu test dispatch PR 50233
  • Unify SFPU test dispatch PR 50602
  • Add unpack_reduce_col_tilizeA_strided Quasar performance test PR 50594
  • Reconfig-escape / empty-uninit sweep for experimental LLKs PR 50563
  • Cover experimental mul_reduce_scalar LLK (Blackhole) PR 50547
  • Deepen reduce_block_max_row coverage (standalone + runtime/reinit variants) PR 50565
  • Add Quasar unary and binary SFPU performance tests PR 51325
  • Remove dead unpacker writes of INT8 / SrcB word-1 config bits PR 49953
  • Performance/precision: atan/asin/acos (fp32) PR 50112
  • Improve ttnn.pow fp32 accuracy for non-integer exponents (<= 3 ULP, integers bit-exact) PR 49649
  • SFPU per-op self-contained init (phase 1 pattern, BH+WH LLK) PR 49529
  • Complete destination-register bank-switching coverage PR 49830
  • Add synchronization scheme doxygen warning PR 50478
  • Extend ckernel::load_blocking to support 16-bit operands PR 50482
  • quasar: pack untilize tiny tiles PR 49850
  • Merge SFPU accuracy and perf into one harness PR 49229
  • Document --mode selector in accuracy tests docs PR 50607
  • Add Quasar performance test skill PR 50556
  • Feature/Testing - Quasar Int8→Int32 reduce support and tests PR 49390
  • Add Quasar LLK perf matmul coverage PR 49365
  • Reduce polygamma exact-summation terms 11 -> 6 (3-5x faster, no accuracy loss) PR 50534
  • Implement support for changing tile size in tilize op PR 50129
  • Add --speed-of-light support to LLK test runners PR 50653
  • Add eltwise_binary Quasar performance test PR 50584
  • quasar sfpu: add negative op (end to end) PR 50752
  • Add uint32 remainder tensor-scalar support PR 49998
  • #47381: reconfig_data_format — always derive int8 state + add SrcOrder overload PR 49119
  • Skrsmanovic/unpack tilize fp8 perf feature PR 49473
  • quasar sfpu: non-approx exp + approx-mode test scaffold PR 50858
  • quasar sfpu: add clamp op (end to end) PR 50753
  • use new functionalities from sfpi PR 50402
  • Revert #49473: restore Blackhole FP8_E4M3 tilize protocol PR 50989
  • quasar sfpu: non-approx reciprocal (Newton-Raphson) PR 50887
  • Convert Blackhole complex SFPU kernels to SFPI PR 49926
  • quasar sfpu: add softplus op (end to end) PR 50754
  • quasar: unpack tilize tiny tiles PR 50640
  • quasar sfpu: non-approx rsqrt (SQRT_23-bits) PR 50891
  • Roll back Blackhole SFPI comparison kernel rewrites PR 51097
  • fix: remove keywords from repo PR 51022
  • Move ALU_FORMAT_SPEC override zero-init to the consuming threads PR 49973
  • Route SFPU dispatch through checked macros PR 50875
  • Add unpack_unary_operand Quasar performance test PR 50596
  • Add eltwise_binary_broadcast Quasar performance test PR 50585
  • Add pack Quasar performance test PR 50588
  • Add pack_l1_acc Quasar performance test PR 50589
  • Add eltwise_unary_datacopy Quasar performance test PR 50587
  • Add eltwise_binary_reuse_dest Quasar performance test PR 50586
  • Add transpose_dest Quasar performance test PR 50592
  • Add pack_untilize Quasar performance test PR 50590
  • Add generated trigonometry SFPU kernel for Quasar PR 43031
  • Add reduce Quasar performance test PR 50591
  • feat: chunk device transfers to speed up llk tests PR 51182
  • Add unary_broadcast Quasar performance test PR 50593
  • Add quant/requant/dequant int32 SFPU kernels PR 48208
  • feat: add llk deterministic tests, checking determinism between runs PR 51192
  • feat: add Quasar SFPU non-approx gelu kernel PR 51532
  • Fix 32-bit unpack-to-dest routing: per-operand unpack-to-dest, drop kernel-wide unpack_to_dest_en PR 49797
  • Update ttsim version to 1.9.7 PR 51564
  • Add unpack_tilize Quasar performance test PR 50595
  • pack_untilize tiny tiles compute api bringup PR 51282
  • Apply MX quantization roundtrip in UntilizeGolden for MxInt formats PR 49200
  • quasar synchronization with semaphores PR 51324
  • [Cleanup] Centralize perf-CSV header construction PR 51483
  • Add perf-parameter-impact skill for analyzing LLK performance reports PR 51681
  • add pack_matmul kernel to fuser PR 51731
  • Move the remaining topk_xl headers into experimental PR 51837

Metalium (tt-metal core)

  • NOC-aware multicast: undo the NOC1 coordinate swap in __emule_multicast_write PR 50244
  • Bound JIT compile fan-out to the fd limit (fix EMFILE on large mesh programs) PR 50330
  • [Bug fix] Fix NoC wedge in watcher sanitize tests under SD mode PR 50176
  • Reset DPRINT state when reasserting Quasar TRISC reset PR 50429
  • Fix iDMA tests hang PR 50301
  • SFPU init: consolidate common+specific per-op, drop the hoist (#50381) PR 50399
  • Remove UInt32 from Qsr compute api PR 50471
  • add fixes to pipeline builder to handle forking PR 50484
  • Fix data race in multicast inbound_kernel: block until L1 tile lands before cb_push_back PR 50183
  • Barrier the loop-back read before reusing the pad-align scratch (#50365) PR 50366
  • Force RISC store to land before the NoC reads it (50367) PR 50368
  • emule: make ASAN sanitizer state per-fiber + fix deferred-mesh snapshot UAF PR 50120
  • Don't assign txn id 0 to DFB implicit sync path PR 50520
  • emule: skip RISC-V kernel link for Emule programs PR 50646
  • Add Guard against mixed API usage of Metal 2.0 and the legacy runtime argument API PR 50483
  • quasar sfpu: shared foundation (config_hash + LREG11 reload) PR 50751
  • Fix Host tensor operations dropping attribtues PR 50285
  • Move pack relu config from pack init to pack hw config PR 50849
  • Fix profiler_data_buffer data layout PR 50770
  • Add legality checks to incoherent Metal 2.0 x legacy API cross-use PR 50954
  • Fix profiler read when ttnn.device.setup_fast_dispatch is used PR 51064
  • Route fabric generated artifacts through TT_METAL_LOGS_PATH PR 51059
  • widen ring_size field for compute side dfb structs to address available l1 in quasar PR 50800
  • Fix ttnn.all_gather Fabric_2D correctness bugs PR 51035
  • Fix DFB tile-counter staging overflow in dataflow_buffer_init PR 51283
  • #51287: Fix heap buffer underflow in memory::pack_from_segments with contentless leading segment PR 51289
  • Cleanup: Remove UBB motherboard check from physical system discovery PR 51202
  • Adjust Quasar dispatch engine changes due to BH prefill Perf Regression PR 51342
  • Skip finish_profiling in invalid firmware iterations PR 51292
  • Reject DRAM tensor binding tokens in LocalTensorAccessor PR 51357
  • Check for reset buffer magic in device simulation PR 51452
  • split user txn id pool from set of txn ids that dfbs use PR 51139
  • Fix failing reduce tests - keep LoFi for Maxpool and pass in tensor_shape to reduce_tile PR 51166
  • Handle unsupported dispatch microbenchmark modes cleanly PR 51518
  • isclose: compare Inf/NaN bit patterns as vInt, not float magnitudes PR 51537
  • Keep config-reg print test writes to supported fields PR 51488
  • Bump ttsim version to v1.9.8 PR 51641
  • emule: fix __emule_fabric_route_follow key truncation (uint32 -> uint64) PR 51696
  • Fixing Quasar DE assignment so related FD kernels are always co-located PR 51740
  • Per-core allocation: one address API, per (device, core) PR 51500
  • MemoryConfig: include per_core_allocation in to_json, operator== and reflection attributes PR 51501
  • Fabric: fix routing on collapsed torus dimensions PR 50946
  • Validate dynamic trace allocations across live traces PR 50503
  • Skip NOC debugging tests before slow-dispatch setup PR 51531
  • Reject DRAM sharded buffers with bank-aliasing shard grids PR 51542
  • Workaround upper 16 TCs corrupting mirrored TCs by remapping upper TCs to an unused shadow PR 51873
  • Improve DPRINT test teardown PR 50433
  • DFB: uplift remaining legacy base-sweep tests to Metal 2.0 PR 50492
  • Runtime Test: add JIT-build compile-throughput CI microbenchmark PR 50296
  • Add Quasar cache-write performance test (test 912): uncached port vs L2 cache+flush PR 50898
  • Add a test runner for deployment tests PR 50914
  • Fix test_clean_init loopback runtime argument ordering PR 51121
  • Update matmul multicast test kernels for CircularBuffer AddrSelector API PR 51025
  • Use explicit host tensor spec conversion in spec-preservation tests PR 51471
  • Add regression tests for sync_build_steps drain-on-throw PR 51032
  • Updating Quasar regression tests to include FD; Enabling more tests to run FD PR 51403
  • emule fiber: rename read_latency primitive to defer_to_quiescence PR 50247
  • Support num_links: max/all in fabric tests, resolved to platform max PR 49851
  • Always-dual tensor prefetcher with GCB-subset PREFETCH routing PR 49863
  • binary_ng: re-apply program-cache-hit args via override_runtime_arguments (fixes SDXL in-place PCC) PR 49828
  • Metal 2.0 MakeMeshWorkloadFromSpecs PR 49437
  • Performance/precision: atan/asin/acos (fp32) PR 50112
  • Add APIs that allow mutating DFB rd/wr pointers on 1xx PR 49971
  • Allow DFB instance multi-binding (on Gen1 only) in Metal 2.0 PR 50355
  • SFPU per-op self-contained init (phase 1 pattern, BH+WH LLK) PR 49529
  • Add op_block Compute API wrappers around op_tile PR 49070
  • Automated UMD Bump 17.07.2026 PR 50190
  • New ttnn.all_gather PR 48301
  • remote JIT compile: preprocess-and-ship with reusable cache PR 49884
  • Auto-resolve destination hops from global mesh shape; unify deadlock … PR 49855
  • Consolidate and rename the DRAM-sender GCB / tensor-prefetcher API PR 49893
  • Bump tracy submodule to tracy#40 PR 50480
  • Bringup basic profiler on quasar PR 49417
  • Use mock env in MetalEnv fork-safety tests to fix CHIP_IN_USE PCIe lock deadlock PR 50147
  • Automated UMD Bump 20.07.2026 PR 50436
  • Fix incorrect helper method names PR 50526
  • Enable support for 2 command queues on Quasar fast dispatch PR 50280
  • Feature/Testing - Quasar Int8→Int32 reduce support and tests PR 49390
  • Forbid Metal 2.0 kernel bindings on non-Metal 2.0 Programs PR 50619
  • Add per-device optimal DRAM worker assignment on MeshDevice (Feature) PR 49783
  • Move tt_metal-only DRAM-sender GCB accessors out of the public API PR 49939
  • Migrate ttnn tensor infrastructure to ttnn namespace PR 50642
  • Bump transformers to 5.12.1 PR 50434
  • Reduce polygamma exact-summation terms 11 -> 6 (3-5x faster, no accuracy loss) PR 50534
  • 4x4 subtorus mesh support for CCLs PR 49881
  • Round Robin Intermesh Routing Contention Resolution PR 49447
  • Fix SD->FD->SD toggle segfault on mock/emulated devices PR 50664
  • Add to_tensor_spec to HostTensor facility PR 50179
  • quasar sfpu: add negative op (end to end) PR 50752
  • Add Quasar fast_tilize wrappers forwarding to plain unpack_tilize path PR 50825
  • remove deprecated global-namespace CoreCoord alias PR 50779
  • Add uint32 remainder tensor-scalar support PR 49998
  • Uplift Pillow and GitPython PR 50702
  • #47381: reconfig_data_format — always derive int8 state + add SrcOrder overload PR 49119
  • Fix coordinate for 2nd DRAM channel in QSR PR 47730
  • quasar sfpu: non-approx exp + approx-mode test scaffold PR 50858
  • Revert pad/unpad checks introduced in #50285 PR 50936
  • quasar sfpu: add clamp op (end to end) PR 50753
  • quasar sfpu: add softplus op (end to end) PR 50754
  • quasar sfpu: non-approx rsqrt (SQRT_23-bits) PR 50891
  • Enable usage of dispatch engine in Quasar PR 48929
  • Remove enqueue-invariance from the Metal 2.0 host API PR 50966
  • Bringup DeviceZoneScopedN on Quasar PR 50900
  • Fix tt_memory DISCRETE load applying segment permutation twice PR 49053
  • Allow explicit TT_METAL_DRAM_BACKED_CQ override on Quasar simulator PR 50469
  • (Bug fix) Re-add TT_METAL_ENABLE_BLACKHOLE_DRAM_PROGRAMMABLE_CORES as force-disable override PR 51071
  • New delivery architecture for the real-time profiler PR 49305
  • fix: remove keywords from repo PR 51022
  • Re-enable runtime unit tests fixed since #44767 PR 50778
  • Add fast_untilize test coverage, verify Quasar forwarding PR 50837
  • Fabric: use stranded Mesh/Torus slots for worker injection PR 50876
  • Revert to_row_major_layout physical-shape check from #50285 PR 51058
  • Feature: emulation-runtime support for host-facing sockets PR 51108
  • Surface reduction for Runtime Tensor specs PR 47732
  • Many-to-many + regex/range patterns for MGD ASIC pinnings PR 51101
  • Bump peft 0.18.1 → 0.19.1 to match transformers 5.12.1 PR 50780
  • Add experimental Blaze-only named kernel-args (blaze_ct_args::/blaze_rt_args:: JIT codegen) PR 48704
  • Fix VIRTUAL_TENSIX_START_X: 1 → 2 to match physical worker origin PR 49290
  • Migrate off deprecated pack_tile_block / copy_block_matmul_partials PR 50978
  • kv chunk table merging & layer ack for PP PR 48826
  • Add Metal 2.0 relaxation-aware TensorSpec hash PR 51043
  • Add back device kernel time columns in perf report on quasar PR 51425
  • #46346: remove legacy matmul init API (mm_init family) PR 50570
  • feat: add Quasar SFPU non-approx gelu kernel PR 51532
  • Bump ttsim version to v1.9.7 PR 51545
  • Rename Metal 2.0 kernel-side binding tokens to *BindingToken PR 51553
  • Fix 32-bit unpack-to-dest routing: per-operand unpack-to-dest, drop kernel-wide unpack_to_dest_en PR 49797
  • Automated UMD Bump 28.07.2026 PR 51339
  • pack_untilize tiny tiles compute api bringup PR 51282
  • Split non-Runtime-Tensor APIs out of experimental/tensor PR 50628
  • Unify big-mesh / multi-mesh fabric test YAMLs on high-level patterns PR 50144
  • Fix multi-op fabric-routing staleness + MCAST barrier direction PR 51366
  • Bringup DeviceTimestampedData + DeviceRecordEvent on quasar PR 51513
  • feat: enhance error logs on quasar wrt hw faults PR 51536
  • Deleting old duplicated tests PR 51623
  • QueueId moved from tt-metal to ttnn PR 51427
  • Expose MGD-inferred fabric type to Python (get_all_mgd_fabric_types) PR 51509
  • Automated UMD Bump 30.07.2026 PR 51575
  • Add Blocked Access Pattern To DFB PR 47589
  • MeshDeviceFixture simplification PR 51644
  • Revert "[UMD Bump] Automated UMD Bump 30.07.2026" PR 51766
  • Cleanup: Consolidate DispatchMemMap to a single canonical instance PR 41687
  • Clean up unpack_reduce_tilizeA_B_init and use hw_startup + init PR 51234
  • Runtime Test: add op-to-op latency CI microbenchmark (Runtime Performance Tests pipeline) PR 49771
  • Move internal Runtime tensor utilities out of public APIs PR 51546
  • Revert "[Feature] Add Blocked Access Pattern To DFB (#47588)" PR 51797
  • fabric: make trimmed speedy credits packet-size aware PR 51154
  • Improve dfb initialization time PR 50476
  • Move the remaining topk_xl headers into experimental PR 51837
  • Sanjaysundaram/sparse multicast edits PR 49436
  • Performance: walk the interleaved bank cycle in the dispatcher's paged write loop PR 51812
  • Feature: Support tensor prefetcher with mcast-in0 matmul PR 50681
  • Change CBReaderWithManualRelease to use page based reading PR 51985

TT-NN

  • Feature: Add single-device fused moe_compute support PR 49886
  • all-gather-minimal-matmul: add chunks and dim to program-cache key PR 50291
  • Fix OOB L1 write in distributed layernorm 2D pre-all-gather merge CB (c_15) PR 50288
  • minimal_matmul: rename local copy_block to fix ambiguous overload (#50386) PR 50390
  • Fix cross-route data race in all_to_all_combine completion credit (Ring) PR 50273
  • Fix WAR in multi-core topk local writer (barrier before CB slot reuse) PR 50318
  • Fix latent deadlock in multi-core reduce_all argmax (exact-match start_sem wait -> wait_min) PR 50315
  • Fix undrained readiness atomic in fused update-cache writer (share_cache) [#48586 item 8] PR 50325
  • Fix receiver store-before-signal L1 ordering race (#50304) PR 50305
  • Force pad-fill stores to land before the loop-back read of cb_pad (#50369) PR 50370
  • Invalidate L1 cache in tile-layout poll loop (#50310) PR 50311
  • Drain index-fill stores before the NoC write reads them (#50375) PR 50376
  • Unify program-cache hashing for Quasar BinaryNgDeviceOperation PR 50017
  • Gate sparse_sdpa_msa block-cyclic multidevice test to Blackhole PR 50613
  • Barrier inbound read before tt_memmove consumers (#50308) PR 50309
  • Flush granular output writes before releasing the CB slot (#50363) PR 50364
  • Fix WAR on source_buffer in row-major interleaved reshape PR 50324
  • Fix latent deadlock in DRAM multi-core sort (split cores->coordinator semaphore) PR 50313
  • ttnn: neighbor_pad_async - set runtime args for no-work cores (fix stale args) PR 50473
  • Fix sparse SDPA FP8 unpack reconfiguration PR 50614
  • Drain first-line fill before loop-back replication read (#50373) PR 50374
  • Use fp32 for QK-scores in SDPA PR 49948
  • Enable UnpackToDestFp32 in matmul when appropriate PR 50250
  • Close fp32-dest-acc auto-promotion gaps in reduce_scatter ops PR 50660
  • Fix writer_l1 flush for non-posted default-path writes (#50306) PR 50307
  • Commit local reduction contribution before release in all_reduce_create_qkv_heads PR 50797
  • Flush per-packet payload before reusing single-slot packet_cb PR 50813
  • Fix WAR on output_l1 mapping scratch in moe_expert_token_remap writer PR 50785
  • Gate tt_memmove NoC self-copy on non-overlap (#50726) PR 50727
  • Fix cross-batch out_ready_sem reset race in strided_reduce_scatter reader PR 50794
  • slice: fix height-sharded RM cache-hit descriptor rebuild (resnet-T3K regression) PR 50894
  • Add missing read barrier before crediting the bilinear reader's CB PR 50981
  • Fix incorrect output and crashes for non-tile-aligned total token counts PR 50934
  • Fix MiniMax indexer perf dtype PR 51012
  • Store welford cb_combined tile through volatile pointer PR 51029
  • Fix sparse SDPA BF16 alignment assertion PR 51136
  • Fix ttnn.all_gather Fabric_2D correctness bugs PR 51035
  • Drain mcast-ready atomics before the in0 receiver returns PR 50997
  • Fix undefined cb_id in Quasar-DM reduce-scaler flush (use dfb_id) PR 51024
  • #50748: Include c_1 staging CB in tilize block factory's L1 budget PR 50551
  • Reland MoE gate typecast removal with bf16-robust gate test (relands #50606) PR 51009
  • ROW_MAJOR ttnn.typecast PR 50545
  • fix(data_movement): correct operator-precedence bug in repeat_program_factory_last_dim PR 51111
  • Limit transpose kernels to active cores PR 50410
  • Incorrect output in height-sharded 1D matmul with multiple row/col blocks PR 50975
  • batch_norm: add output CB to UnpackToDestFp32 list for typecast path PR 51313
  • Fix Fabric_2D hang in ttnn.all_gather unicast kernels PR 51422
  • Bug fix: bcast ShardedHOptimised latent buffer over-runs PR 51056
  • Reject wrong combination for fused_rms_minimal op PR 51077
  • Support 1D tensors in all_reduce op PR 51347
  • Ccl collective determinism PR 50688
  • Fix checks in all_gather PR 51239
  • #51133: Refuse per-core allocation instead of silently downgrading it PR 51430
  • Fix use after free for MemoryConfig in move op PR 51535
  • binary_ng: re-enable ROW_A_COL_B mixed broadcast on the DFB path (craq-sim #205 fixed) PR 51541
  • Fix second-stage-reader CB credit/layout in pre_allgather receiver (defensive) PR 50815
  • rotary_embedding: fix decode cos/sin freeze on program-cache hit PR 49889
  • Drain tree-add stores before signaling the parent (#50371) PR 50372
  • isclose: compare Inf/NaN bit patterns as vInt, not float magnitudes PR 51537
  • Let the generic op accept per-core allocated tensors PR 51573
  • #47299: Enable native path for RM HEIGHT-sharded sub-NoC-aligned sticks in TM ops PR 50877
  • enable index fill output specs PR 49999
  • Fix batch_norm running_statistics drain hang and three latent Metal 2.0 porting bugs PR 51494
  • Per-core allocation: one address API, per (device, core) PR 51500
  • MemoryConfig: include per_core_allocation in to_json, operator== and reflection attributes PR 51501
  • Fix folded FABRIC_2D CCL startup barriers PR 51869
  • Make sampling and top-k deterministic on sub-core grids. PR 50687
  • Reject DRAM sharded buffers with bank-aliasing shard grids PR 51542
  • Move sparse_sdpa / indexer_score perf checks to the realtime device profiler PR 49878
  • Disable retile 1x32 unit test PR 50838
  • Fix indexer_score TP1 performance targets PR 51147
  • Use pytest-managed temp paths for multi-device as_tensor cache tests PR 51595
  • sdpa_decode: drop redundant custom compute_program_hash PR 49872
  • inbound_socket_service_sync: drop redundant custom compute_program_hash PR 49873
  • create_qkv_heads: drop redundant custom compute_program_hash PR 49868
  • layernorm: honour two missing CB contracts in the rm_gb reader PR 49974
  • Always-dual tensor prefetcher with GCB-subset PREFETCH routing PR 49863
  • Optimizations for Qwen3.6-27B PR 48861
  • binary_ng: re-apply program-cache-hit args via override_runtime_arguments (fixes SDXL in-place PCC) PR 49828
  • Performance/precision: atan/asin/acos (fp32) PR 50112
  • Add APIs that allow mutating DFB rd/wr pointers on 1xx PR 49971
  • Improve ttnn.pow fp32 accuracy for non-integer exponents (<= 3 ULP, integers bit-exact) PR 49649
  • ds_prefill - Fuse tilize with unified routed expert for Blackhole PR 49744
  • New ttnn.all_gather PR 48301
  • sparse_sdpa_msa: add in-kernel block-cyclic KV remap PR 49490
  • bounded KV cache + chunked prefill to support 256K ISL [Feature][Optimization] PR 49614
  • Guard optional-arg crashes and document the layout contract PR 49864
  • DIspatch row major path refactor + enable dispatch to accept scale tensor as additional input PR 48694
  • Consolidate and rename the DRAM-sender GCB / tensor-prefetcher API PR 49893
  • eltwise/unary: replace get_dynamic_runtime_args with override_runtime_arguments PR 50335
  • Remove mesh_device_ stub from ttnn::Tensor PR 47762
  • moreh_adamw: replace get_dynamic_runtime_args with override_runtime_arguments PR 50339
  • Revert "[GroupNorm][Welford] Fix receiver store-before-signal L1 ordering race (#50305)" PR 50553
  • indexer_score: consolidate seq-shard axes into a single seq_shard_axes arg PR 50004
  • Fix incorrect helper method names PR 50526
  • unified_routed_expert_ffn: expert-bias support (gpt-oss) PR 49619
  • Conv2D default hash. PR 48839
  • Use default hash for BinaryNg PR 48755
  • rand: replace get_dynamic_runtime_args with override_runtime_arguments PR 50488
  • Fix comp_pcc false pass for constant tensors PR 42848
  • #49283: Make ttnn.split consistent with torch PR 49575
  • Add per-device optimal DRAM worker assignment on MeshDevice (Feature) PR 49783
  • Migrate ttnn tensor infrastructure to ttnn namespace PR 50642
  • Reduce polygamma exact-summation terms 11 -> 6 (3-5x faster, no accuracy loss) PR 50534
  • Implement support for changing tile size in tilize op PR 50129
  • #45821: migrate AllToAllCombineDeviceOperation to default program-cache hash PR 50015
  • #45821: migrate BroadcastDeviceOperation to default program-cache hash PR 50018
  • Remove deprecated use_legacy flag from ttnn.std/ttnn.var PR 50527
  • Fix indexer score pack API deprecation PR 50691
  • data_movement/slice: replace get_dynamic_runtime_args with override_runtime_arguments PR 50347
  • permute: drop redundant custom compute_program_hash PR 49866
  • moreh_adam: replace get_dynamic_runtime_args with override_runtime_arguments PR 50346
  • Bug fix: preserve dtypes in repeat_interleave PR 44115
  • Fix ternary op silent-garbage on program-cache hit PR 49751
  • Add realtime perf checks for topk_large_indices PR 49673
  • Ideogram 4.0 PR 49923
  • outbound_socket_service_sync: drop redundant custom compute_program_hash PR 49874
  • data_movement/move: replace get_dynamic_runtime_args with override_runtime_arguments PR 50349
  • data_movement/tilize: replace get_dynamic_runtime_args with override_runtime_arguments PR 50342
  • remove deprecated global-namespace CoreCoord alias PR 50779
  • Add uint32 remainder tensor-scalar support PR 49998
  • Equal-Count Welford Reduction PR 50624
  • Remove typecasts from DEVICE_FP32 MoE gate PR 50606
  • Keep indexer Q in BFP8 through scoring PR 50627
  • #47381: reconfig_data_format — always derive int8 state + add SrcOrder overload PR 49119
  • DeepSeek: Add mixed-format KV cache for sparse MLA PR 50207
  • ds_prefill(moe_gate) - revert #50606 (typecast removal broke MoE gate PCC) PR 50902
  • Expose PagedCacheGeometryOverride on chunked/paged SDPA APIs PR 50437
  • ds_prefill(single_routed_expert_tests) - refactor into ISL sweep tests PR 50464
  • Universal input support for untilize_with_unpadding PR 50383
  • Metal 2.0 spec factory: rename MetalV2FactoryConcept + add CustomProgramSpecFactoryConcept PR 50942
  • pipeline-parallel (multi-galaxy D2D) prefill PR 50466
  • sparse_sdpa: replace get_dynamic_runtime_args with override_runtime_arguments PR 50351
  • Performance: Migrate reduce_scatter_async_minimal (Ring) to Mux v2 PR 50644
  • New delivery architecture for the real-time profiler PR 49305
  • ds_prefill - Routed expert optimization removing unnecessary zero writing to L1 memory PR 50885
  • Fix strided input with 8 bit data format on BH. PR 49792
  • Ring-fused indexer_score: overlap the SP all-gather with scoring PR 49899
  • ternary: in-place override_runtime_arguments (fix get_dynamic-migration cache-hit regression) PR 50759
  • Remove dead code from data_movement/sort/device/kernels/dataflow/writer_cross_core_data_exchange.cpp PR 51063
  • deepseek_prefill: per_token_cast_to_fp8 TILE input layout PR 50741
  • Adapt masked_bincount to accept TILE interleaved expert_indices PR 50645
  • Migrate MM/Fused/Reduce Kernels from CircularBuffer to DataflowBuffer PR 49173
  • ds_prefill - Adaptive per_core_M for routed expert PR 51181
  • Surface reduction for Runtime Tensor specs PR 47732
  • Metal 2.0 port: data_movement/scatter PR 51196
  • Prepare moreh_mean for Device 2.0 port PR 51402
  • port(metal_2.0): experimental/reduction/integral_image → MetalV2FactoryConcept (verified passing) PR 51087
  • Port data_movement/clone factories PR 50903
  • port(metal_2.0): moreh/moreh_fold → MetalV2FactoryConcept (verified passing) PR 51092
  • Accept allocator DRAM bank validation in pad unsupported test PR 49722
  • Use separate DRAM and L1 shard grids in tilize regression test PR 49581
  • Add experimental Blaze-only named kernel-args (blaze_ct_args::/blaze_rt_args:: JIT codegen) PR 48704
  • mla: add 2D all-to-all for GLM sparse attention PR 49786
  • floor_div: collapse the Inf guard into a single isinf check PR 51437
  • Implement custom matmul for decode, M = 1 and input + weights are L1 Width Sharded PR 48422
  • Migrate off deprecated pack_tile_block / copy_block_matmul_partials PR 50978
  • kv chunk table merging & layer ack for PP PR 48826
  • Device 2.0 Port for eltwise_copy kernel PR 51179
  • Port data_movement/fill_rm factory PR 51054
  • binary_ng: subtile-broadcast + tensor-scalar on the Metal 2.0 DFB path PR 50935
  • Add Metal 2.0 relaxation-aware TensorSpec hash PR 51043
  • Guard Rev C slice correction script execution behind main() PR 48729
  • #46346: remove legacy matmul init API (mm_init family) PR 50570
  • port(metal_2.0): moreh/moreh_abs_pow → MetalV2FactoryConcept (verified passing) PR 51090
  • port(metal_2.0): moreh/moreh_sum_backward → MetalV2FactoryConcept (verified passing) PR 51104
  • Port data_movement/fill_pad factories PR 50904
  • Rename Metal 2.0 kernel-side binding tokens to *BindingToken PR 51553
  • uniform: replace get_dynamic_runtime_args with override_runtime_arguments PR 50338
  • experimental/paged_cache fill_cache: replace get_dynamic_runtime_args with override_runtime_arguments PR 50344
  • nlp_create_qkv_heads: replace get_dynamic_runtime_args with override_runtime_arguments PR 50340
  • kv_cache/update_cache: replace get_dynamic_runtime_args with override_runtime_arguments PR 50343
  • paged_cache/fused_update_cache: replace get_dynamic_runtime_args with override_runtime_arguments PR 50350
  • experimental/paged_cache update_cache: replace get_dynamic_runtime_args with override_runtime_arguments PR 50345
  • #45831: rotary_embedding_indexed — trace-safe per-element-tensor metadata overload (op + test, no trace) PR 48903
  • #46577: zero_padded_kv_cache — trace-safe per-element-tensor metadata overload (op + test, no trace) PR 48905
  • Fix 32-bit unpack-to-dest routing: per-operand unpack-to-dest, drop kernel-wide unpack_to_dest_en PR 49797
  • #45832: update_padded_kv_cache — trace-safe per-element-tensor metadata overload (op + test, no trace) PR 48906
  • binary_ng, unary: move override_runtime_arguments from DeviceOperation to the factory PR 50735
  • slice_rm + roll: remove smuggled buffer-address runtime args PR 49549
  • concat: drop redundant custom compute_program_hash PR 49867
  • nlp_create_qkv_heads_boltz: replace get_dynamic_runtime_args with override_runtime_arguments PR 50348
  • mesh_partition: fix height-sharded RM slice cache-hit rebuild PR 50918
  • Revert "[Metal 2.0]: Port data_movement/fill_pad factories (#50904)" PR 51605
  • port(metal_2.0): moreh/moreh_dot → MetalV2FactoryConcept (verified passing) PR 51085
  • #45830: ring_mla — trace-safe per-element-tensor metadata overload (op + test, no trace) PR 48907
  • Split non-Runtime-Tensor APIs out of experimental/tensor PR 50628
  • Metal 2.0 port: data_movement/sort PR 51407
  • #48552: Update Quasar support for ResNet50 and experimental TTNN operations PR 51591
  • port(metal_2.0): moreh/moreh_dot_backward → MetalV2FactoryConcept (verified passing) PR 51091
  • QueueId moved from tt-metal to ttnn PR 51427
  • Metal 2.0 port: data_movement/split PR 51274
  • Metal 2.0 port: data_movement/repeat PR 51068
  • Expose MGD-inferred fabric type to Python (get_all_mgd_fabric_types) PR 51509
  • docs(plus_one): correct dtype/rank and document skip_negative_entries semantics PR 51300
  • Port moreh/moreh_norm (3/3 factories) PR 51479
  • Port moreh/moreh_mean (3/3 factories) PR 51445
  • Ring reduce-scatter: contiguous intermediate PR 50933
  • Metal 2.0 port: reduction/accumulation PR 51614
  • Optimize cached height-sharded transpose dispatch & #48928 performance PR 51685
  • interleaved_to_sharded_partial: stop over-keying on slice_index PR 49890
  • Port Typecast to Metal 2.0 PR 51397
  • Clean up unpack_reduce_tilizeA_B_init and use hw_startup + init PR 51234
  • #51744: Fix Quasar data-movement builds using common memmove helpers PR 51746
  • Metal 2.0 port: data_movement/indexed_fill PR 51039
  • Move internal Runtime tensor utilities out of public APIs PR 51546
  • Conv2d: use CB FIFO semantics for partial accumulation PR 51664
  • Move the remaining topk_xl headers into experimental PR 51837
  • Add GPT-OSS sliding prefill to RingJointSDPA PR 51438
  • Metal 2.0 port: full PR 51796
  • Feature: Support tensor prefetcher with mcast-in0 matmul PR 50681

tt-train

  • minimal_matmul: rename local copy_block to fix ambiguous overload (#50386) PR 50390
  • Removing Reshape in Qwen3 Attention PR 50302
  • Migrate ttnn tensor infrastructure to ttnn namespace PR 50642
  • bug fix - Allow target_modules to be empty when initializing LoraModel PR 48837
  • Implement VocabParallelEmbedding and FeatureParallelEmbedding PR 49382
  • bug fix - Remove explicit device open and close for mla_fused_sdpa_matches_composite_forward_and_backward test PR 48832
  • #46346: remove legacy matmul init API (mm_init family) PR 50570
  • Qwen3 Fusing KV Projections and Fixing Llama TP PR 50673
  • Move internal Runtime tensor utilities out of public APIs PR 51546

Models

  • fix: add missing MlaKvCaches module (deepseek_v3_d_p prefill runner) PR 50236
  • Fix NameError in test_align_shape_to_tile due to missing import PR 50252
  • Fix Attention1D vLLM token validation PR 49932
  • Bump ViT wormhole/t3000 trace_region_size to fix N300 TRACE OOM PR 50493
  • Fix DRAM leak in shared expert reduce_scatter keepalive PR 50705
  • Fix batched-prefill garbage decode: capture decode trace before prefill trace PR 50706
  • Use fp32 for QK-scores in SDPA PR 49948
  • Enable UnpackToDestFp32 in matmul when appropriate PR 50250
  • fix galaxy prefill dispatch core-grid assert (num_untilizers_per_sender=1) PR 50710
  • Fix stale MLA KV-cache callers after #50207 wrapper migration PR 50986
  • Gemma4 sliding-tail dealloc in chunked prefill. PR 51041
  • Qwen3-VL: slice page table before multimodal prefill PR 51148
  • Reland MoE gate typecast removal with bf16-robust gate test (relands #50606) PR 51009
  • Qwen3.6: load text weights via explicit HF classes, not AutoModelForCausalLM PR 51327
  • Fix MoE padding config under rotated chunked prefill PR 51440
  • Llama Galaxy sampling: per-request seeding, prefill logit routing, and mixed greedy/random decode PR 50685
  • Ccl collective determinism PR 50688
  • Fix Python 3.10 support in experimental pipelines PR 51404
  • Per-core allocation: one address API, per (device, core) PR 51500
  • Fix TT-DiT persistent reduce-scatter buffer shape PR 51774
  • Fix the mock-migration deadlock, make the CI leg readable PR 51693
  • Make sampling and top-k deterministic on sub-core grids. PR 50687
  • Fix DeepSeek Ring reduce-scatter intermediate shape PR 51916
  • tt-transformers: Fix garbled token outputs on P100 PR 50792
  • Bumping up pytest timeout for Blaze - Prefill Block Determinism test PR 50231
  • Jameslee/fix wan2 2 i2v unit test PR 50295
  • GLM DSA: record indexer-K cache PCC in chunked prefill tests PR 49930
  • #51364: add isolated ops tests for quasar llama PR 51368
  • validate producer/runner KV against a prompt-generated reference PR 50873
  • Recalibrate prefill block perf target PR 51463
  • Add rotated/padded chunked-prefill CI coverage (mid15k padded test + Blaze wiring) PR 51466
  • GDN prefill: native depthwise conv1d (replaces manual FIR) PR 49946
  • Optimizations for Qwen3.6-27B PR 48861
  • Add janus pro vision embedding PR 47801
  • prefill perf harness: whole-sequence + last-chunk timing PR 50327
  • Recalibrate MoE galaxy pad50 device-perf baseline PR 50332
  • sparse_mla: fold single-shot onto the block-cyclic path (persist indexer cache) PR 49719
  • ds_prefill - Fuse tilize with unified routed expert for Blackhole PR 49744
  • Wan2.2: support per-request flow_shift, boundary_ratio, and updatable guidance_scale PR 47265
  • Add token-matching accuracy test for Qwen3-32B on Galaxy PR 49737
  • WAN pipeline: emit per-step progress events + fix module loaded flag PR 47715
  • bounded KV cache + chunked prefill to support 256K ISL [Feature][Optimization] PR 49614
  • ci: migrate sparse/DSA MLA + GLM prefill block tests to fabric2d PR 50221
  • Add multihost (exabox) model unit-test job, selected by SKU PR 50151
  • batch=8 support for Qwen3.6-27B PR 48380
  • DIspatch row major path refactor + enable dispatch to accept scale tensor as additional input PR 48694
  • ds_prefill(cleanup) - remove routed expert multichip test PR 50441
  • indexer_score: consolidate seq-shard axes into a single seq_shard_axes arg PR 50004
  • Sparse MLA perf harness: Tracy → realtime profiler PR 50122
  • glm chunked prefill: e2e perf timing + padding-correct populated-KV gather PR 50011
  • TTTv2 Llama-3.2-1B: pending CI cases + batched prefill + fresh perf gates PR 49292
  • Move SDXL tests to Tier 2 CI PR 50006
  • unified_routed_expert_ffn: expert-bias support (gpt-oss) PR 49619
  • Parametrize k_chunk_size for DRAMZeroFill PR 50625
  • Validate index KV cache in prefill_producer PR 50384
  • deepseek_prefill: skip fabric2d mesh-4x2 MoE tests (all-gather hang) PR 50663
  • Unify b=1/b>1 serving for Qwen3.6-27B PR 50524
  • Fix comp_pcc false pass for constant tensors PR 42848
  • DeepSeek V3 prefill: enable X-axis (TP) Ring topology PR 48711
  • prefill: add Kimi-K2.7 model adapter PR 50610
  • Qwen3-32B decoder blackhole galaxy PR 49593
  • Llama-3.2-3B-Instruct → CI parity with TTTv1 + CI-leg substitution PR 50227
  • On-device fast argmax path for Qwen3.6-27B PR 50783
  • Ideogram 4.0 PR 49923
  • outbound_socket_service_sync: drop redundant custom compute_program_hash PR 49874
  • Remove typecasts from DEVICE_FP32 MoE gate PR 50606
  • Add GLM-5.2 chunked-prefill CI jobs and rebalance bh_sc1 timeouts PR 50841
  • prefill: add kimi27 manifest, rename kimi -> kimi26 PR 50863
  • Keep indexer Q in BFP8 through scoring PR 50627
  • MSA/Indexer cache-read: use in-kernel block-cyclic KV remap PR 50851
  • #47381: reconfig_data_format — always derive int8 state + add SrcOrder overload PR 49119
  • Remove throttle from SDXL sanity tests PR 50888
  • DeepSeek: Add mixed-format KV cache for sparse MLA PR 50207
  • ds_prefill(moe_gate) - revert #50606 (typecast removal broke MoE gate PCC) PR 50902
  • Expose PagedCacheGeometryOverride on chunked/paged SDPA APIs PR 50437
  • ds_prefill(single_routed_expert_tests) - refactor into ISL sweep tests PR 50464
  • pipeline-parallel (multi-galaxy D2D) prefill PR 50466
  • ds_prefill - Routed expert optimization removing unnecessary zero writing to L1 memory PR 50885
  • Avoid composite reduce-scatter fallback for indexer wts all-reduce PR 51007
  • Kimi Prefill: Dflash support on 1 Galaxy PR 50235
  • glm indexer: load wq_b/wk (Q/K proj) as bf8 instead of bf16 PR 51005
  • Parallel (sharded) token embedding PR 50170
  • Add prefill runner to Blaze Models Prefill tests and rebalance CI budget time PR 51013
  • Publish Blaze prefill performance summaries PR 51055
  • Adapt masked_bincount to accept TILE interleaved expert_indices PR 50645
  • #51177: Add an experimental Quasar port of Llama 3.2 1B PR 51337
  • Blaze prefill (route perf jobs to high-power galaxy pool) PR 51187
  • Revert "deepseek_prefill: skip fabric2d mesh-4x2 MoE tests (all-gathe… PR 51328
  • CODEOWNERS: add gpt-oss prefill owners PR 50768
  • ds_prefill - Adaptive per_core_M for routed expert PR 51181
  • GPT-OSS prefill: package scaffold + attention (GQA/sinks/sliding) PR 50223
  • mla: add 2D all-to-all for GLM sparse attention PR 49786
  • kv chunk table merging & layer ack for PP PR 48826
  • ci: run GLM prefill tests as GLM-5.2 instead of GLM-5.1 PR 51183
  • #51472: add quasar ops tests that exercise ops/inputs that would be exercised by yolo8 PR 51478
  • Register MiniMax-M3 Tier 1 tests and model targets PR 50977
  • GPT-OSS prefill: chunked KV cache + indexed RoPE PR 50228
  • LTX-2.3: on-device LoRA bind (skip host re-fuse + full-weight reload) PR 49190
  • #45831: rotary_embedding_indexed — trace-safe per-element-tensor metadata overload (op + test, no trace) PR 48903
  • #46577: zero_padded_kv_cache — trace-safe per-element-tensor metadata overload (op + test, no trace) PR 48905
  • #45832: update_padded_kv_cache — trace-safe per-element-tensor metadata overload (op + test, no trace) PR 48906
  • #48552: Update Quasar support for ResNet50 and experimental TTNN operations PR 51591
  • Recalibrate perf baseline PR 51625
  • Adding MoE gate module tests for Minimax m2.7/ GPT-OSS / GLM 5.1 prefill PR 49468
  • fix kvpe_cache kwarg at kv chunk table call sites PR 51684
  • prefill_producer: arbitrary slot→slot migrations + per-slot prompts PR 50809
  • Conv2d: use CB FIFO semantics for partial accumulation PR 51664
  • GPT-OSS prefill: MoE wrapper (router + fused experts, SwiGLU-OAI) PR 50254
  • Add BH GLX Wan2.2 tests to CI PR 50939
  • Change test_dispatch_combine_perf baselines from 25k single-shot to 5k chunked captured picks; DSv3, Kimi2.6, GLM5.2 PR 51426
  • GPT-OSS prefill: model + runtime + common/prefill adapter PR 50265
  • chunked prefill: SP ring cache-read (sinks on all layers) PR 51992
  • CI-readiness for the 9 remaining TTTv2 models — one branch: e2e-leg substitution, CI-case parity, prefill/decode parity PR 51205

TT-STL

  • Use default hash for BinaryNg PR 48755

Infrastructure & CI

  • Add Guard against mixed API usage of Metal 2.0 and the legacy runtime argument API PR 50483
  • Enable UnpackToDestFp32 in matmul when appropriate PR 50250
  • Fix Blackhole E2E QuietBox time budget PR 50856
  • MINFRA-1264 ttop-delete-allocation: fix dangling allocations + refactor ttop allocation actions PR 50729
  • MINFRA-1297 ttop-create-environment: refactor environment actions onto shared await PR 51079
  • Fix ttnn.all_gather Fabric_2D correctness bugs PR 51035
  • fix(ci): stop anonymous Docker Hub pulls for SBOM attestation scanner PR 51203
  • Retire Blackhole per-test chip-reset/bin-pack workaround now that #45727 fixes the root cause PR 51157
  • fix(ci): post repo-assist add-comment as org-member service account PR 51519
  • #51133: Refuse per-core allocation instead of silently downgrading it PR 51430
  • Fix allowed-files globs to match nested source paths PR 51539
  • Post a PR comment when CodeOwners ping hits a GitHub API rate limit PR 51631
  • Add pinned curl 8.x tool image to fix Garage SigV4 uploads PR 51626
  • Fetch CI logs via the github MCP tool, not the credential-less gh CLI PR 51554
  • ttop-await-resource: don't merge stderr into the polled value; re-mint on any error PR 51647
  • Update DSv3 weights path in CI after deletion PR 51612
  • fix(repo-assist): check for existing linked PRs before opening a fix [skip ci] PR 51811
  • Fix the mock-migration deadlock, make the CI leg readable PR 51693
  • fix(silencer): recompile stale lock file (E009 CONFIG_HASH_MISMATCH) [skip ci] PR 51846
  • Split merge gate tests into 2 groups PR 50450
  • Jameslee/fix wan2 2 i2v unit test PR 50295
  • remove stale Wan2.2 test names/delete tests which have been migrated over PR 50462
  • Move sparse_sdpa / indexer_score perf checks to the realtime device profiler PR 49878
  • Updating tt-metal Quasar test suite PR 50647
  • Unify SFPU test dispatch PR 50602
  • Runtime Test: add JIT-build compile-throughput CI microbenchmark PR 50296
  • Add rotated/padded chunked-prefill CI coverage (mid15k padded test + Blaze wiring) PR 51466
  • Added Fabric DC validation neighbor-exchange tests PR 50638
  • Updating Quasar regression tests to include FD; Enabling more tests to run FD PR 51403
  • Update firmware setup script to support v19.x in both tt-system-firmware and in tt-firmware PR 50241
  • Feature: QoL - add the perf/acc measurements in the CI step PR 49816
  • Optimizations for Qwen3.6-27B PR 48861
  • ci: use ubuntu-slim runner for trivial ubuntu-latest jobs PR 49992
  • Add op_block Compute API wrappers around op_tile PR 49070
  • Improve artifact download on multihost runners PR 50225
  • Remove env vars set in ttop PR 50400
  • ds_prefill - Quick fixes for CI PR 50242
  • New ttnn.all_gather PR 48301
  • Add token-matching accuracy test for Qwen3-32B on Galaxy PR 49737
  • ci: migrate sparse/DSA MLA + GLM prefill block tests to fabric2d PR 50221
  • Add multihost (exabox) model unit-test job, selected by SKU PR 50151
  • batch=8 support for Qwen3.6-27B PR 48380
  • DIspatch row major path refactor + enable dispatch to accept scale tensor as additional input PR 48694
  • ci: revert codeowners post-comment job to ubuntu-latest PR 50461
  • Route LLK PR review workflow egress through Tailscale for allowlisting PR 49187
  • Auto-resolve destination hops from global mesh shape; unify deadlock … PR 49855
  • add nnyamagoudar-TT as fabric test infra codeowner PR 50490
  • Add nnyamagoudar as codeowner of run_fabric_tests.sh PR 50289
  • MINFRA-548: remove 2x/4x BH-LB jobs PR 50504
  • Bringup basic profiler on quasar PR 49417
  • ci: cap Qwen3-VL 32B context on WH-T3K PR 50217
  • Add GitHub Agentic Workflows scaffolding and daily-repo-status workflow PR 50535
  • DCAMP-1042 - Create a minimal Docker image for the cabling generator PR 49919
  • glm chunked prefill: e2e perf timing + padding-correct populated-KV gather PR 50011
  • Add ai-summary to blaze prefill tests PR 50609
  • TTTv2 Llama-3.2-1B: pending CI cases + batched prefill + fresh perf gates PR 49292
  • Move SDXL tests to Tier 2 CI PR 50006
  • Fix all-model-tests startup failure: grant id-token to unit-test callers PR 50623
  • Add TT-Fabric and distributed Copilot review instructions PR 47386
  • Add workflow scanning for runner failures PR 50333
  • Move CPM dependency cache to Garage S3 PR 50379
  • Add missing VBench environment variable PR 50440
  • #50692: Add All Model Tests to CICD data production PR 50693
  • Qwen3-32B decoder blackhole galaxy PR 49593
  • Fix deepseek quad tests PR 50713
  • Runner failure scan - Add workflow run link to runner failure Slack posts PR 50715
  • Increase timeout of nightly L2 ttnn experiment test on P100 PR 50641
  • MINFRA-1077: Read civ2 annotations to set host name into superset PR 49756
  • Auto approve tracy version bumps PR 50491
  • Llama-3.2-3B-Instruct → CI parity with TTTv1 + CI-leg substitution PR 50227
  • ci: fix mpirun CPU-thread throttle in Blaze prefill jobs (--bind-to none + OMP_NUM_THREADS) PR 50740
  • Round Robin Intermesh Routing Contention Resolution PR 49447
  • outbound_socket_service_sync: drop redundant custom compute_program_hash PR 49874
  • Uplift Pillow and GitPython PR 50702
  • Add GLM-5.2 chunked-prefill CI jobs and rebalance bh_sc1 timeouts PR 50841
  • update code owners for the bringup tools PR 50857
  • feat(ci): enable treeless checkout for merge gate and sanity workflows [Performance] PR 50846
  • #47381: reconfig_data_format — always derive int8 state + add SrcOrder overload PR 49119
  • docker: use COPY --link for prebuilt tool/venv layers PR 50890
  • feat(docs): OpenSearch docs search indexing PR 49956
  • DeepSeek: Add mixed-format KV cache for sparse MLA PR 50207
  • Add optional model_family field to models e2e/unit/sweep test registries PR 50919
  • Disable the Llama 8B TP=4 DDP=8 run on Galaxy Perf Workflow PR 50906
  • Add repo-assist agentic workflow PR 50829
  • Handle re-runs and empty summaries PR 50950
  • Shard llk_smoke_blackhole into two pytest groups PR 50951
  • Add DeepWiki MCP tool + guidance and fix 4 triage bugs (supersedes #50962, #50963) [skip ci] PR 50964
  • ci(build): Package the ttnn wheel in-place instead of a duplicate manylinux compile [Performance] PR 50555
  • Enable watcher in Tier 1/2/3 model unit test pipelines PR 48924
  • Runner failure scan improvements PR 51048
  • Test Only: Add Quasar coverage to LLK perf workflow PR 50880
  • New delivery architecture for the real-time profiler PR 49305
  • Add CODEOWNERS for tt_metal/impl/threading PR 51141
  • ci(sanity-tests): use build-inplace-wheel for faster sanity test builds [Performance] PR 50984
  • Kimi Prefill: Dflash support on 1 Galaxy PR 50235
  • ci(merge-gate): switch to in-place wheel build [Performance] PR 51146
  • Use cached HF weights for Galaxy Wan2.2 perf test (#46086) PR 48403
  • Re-enable runtime unit tests fixed since #44767 PR 50778
  • ci: use zstd --long -9 for CPM cache tarball PR 51242
  • Fix auto-approve UMD job failing on bracketed bot usernames (curl URL globbing) [skip ci] PR 51312
  • Add prefill runner to Blaze Models Prefill tests and rebalance CI budget time PR 51013
  • Publish Blaze prefill performance summaries PR 51055
  • #51177: Add an experimental Quasar port of Llama 3.2 1B PR 51337
  • Blaze prefill (route perf jobs to high-power galaxy pool) PR 51187
  • CODEOWNERS: add gpt-oss prefill owners PR 50768
  • GPT-OSS prefill: package scaffold + attention (GQA/sinks/sliding) PR 50223
  • Fix merge queue SFTP upload key PR 51410
  • Many-to-many + regex/range patterns for MGD ASIC pinnings PR 51101
  • kv chunk table merging & layer ack for PP PR 48826
  • ci: run GLM prefill tests as GLM-5.2 instead of GLM-5.1 PR 51183
  • Reduce LLK FD/SD test timeout from 15 to 10 for tests PR 51481
  • Register MiniMax-M3 Tier 1 tests and model targets PR 50977
  • #46346: remove legacy matmul init API (mm_init family) PR 50570
  • feat: add llk deterministic tests, checking determinism between runs PR 51192
  • Add "Silencer" agentic workflow to root-cause CI log noise PR 51434
  • Upgrade gh-aw agentic workflows to v0.84.0 PR 51549
  • ci(workflows): switch to in-place wheel build [Performance] PR 51330
  • Add Copilot review rule for incomplete PR descriptions PR 51577
  • Release notes: list the full PR set (fetchViaCommits) PR 51604
  • pre-commit: forbid ProgramDescriptor rebuilds inside override_runtime_arguments PR 51590
  • Split non-Runtime-Tensor APIs out of experimental/tensor PR 50628
  • Unify big-mesh / multi-mesh fabric test YAMLs on high-level patterns PR 50144
  • produce data: fix epoch timestamps being uploaded to cicd_test PR 51630
  • #1358: Add agentic CI failure triage shadow pilot (gh-aw) PR 51412
  • Add Metal public API Copilot review instructions PR 47259
  • Run repo-assist agentic workflow 4x/day instead of daily [skip ci] PR 51654
  • ttnn sweeps → pipeline-reorg: collapse per-HW caller jobs PR 50430
  • Runtime Test: add op-to-op latency CI microbenchmark (Runtime Performance Tests pipeline) PR 49771
  • Improve runner failure reports with live GitHub jobs [skip ci] PR 51801
  • chore(gh-aw): bump to v0.84.2 and let Silencer dispatch CI on its own PR branches PR 51834
  • Add BH GLX Wan2.2 tests to CI PR 50939
  • Add S3 write+readback test to clang-tidy-reusable PR 45911
  • Fire tt-triage on hang in Blaze Models Prefill tests PR 51679
  • fix(ci): use manylinux wheel for ubuntu 24.04 sanity tests [Bug Fix] PR 51973
  • CI-readiness for the 9 remaining TTTv2 models — one branch: e2e-leg substitution, CI-case parity, prefill/decode parity PR 51205

Documentation

  • Barrier the loop-back read before reusing the pad-align scratch (#50365) PR 50366
  • fix(docs): load shared tt_theme.css from global _static PR 51053
  • Fix mobile docs layout: prevent content from being crushed into a narrow right strip PR 51114
  • Optimizations for Qwen3.6-27B PR 48861
  • Add op_block Compute API wrappers around op_tile PR 49070
  • #47381: reconfig_data_format — always derive int8 state + add SrcOrder overload PR 49119
  • feat(docs): apply Tenstorrent Figma theme PR 49957
  • feat(docs): top nav + Search/Ask-AI modal + feedback widget PR 49959
  • New delivery architecture for the real-time profiler PR 49305
  • #46346: remove legacy matmul init API (mm_init family) PR 50570
  • docs(nav): add Galaxy systems and Cloud-Native Support to top nav PR 51769
  • Clean up unpack_reduce_tilizeA_B_init and use hw_startup + init PR 51234

Tooling

  • Fix output width in mpi case PR 50238
  • Allow simple table borders PR 50249
  • Disable dumping watcher ringbuffer when watcher is not enabled PR 50406
  • Added Fabric DC validation neighbor-exchange tests PR 50638
  • Replace mdashes with regular dashes PR 50234
  • Support num_links: max/all in fabric tests, resolved to platform max PR 49851
  • Feature: QoL - add the perf/acc measurements in the CI step PR 49816
  • tag every recover.sh / run_validation.sh output line with [hostname][time] PR 50256
  • Gate elf cache stats PR 50407
  • Surface skipped scripts PR 50426
  • #DIIM-247: recover.sh and run_validation.sh optimizations and new features PR 50392
  • Bringup basic profiler on quasar PR 49417
  • Migrate ttnn tensor infrastructure to ttnn namespace PR 50642
  • Feature Addition To CablingGenerator for Grouping+Filtering PR 49554
  • Add resolve_host_ring_order.py PR 50130
  • DIIM-260 - Allow Exabox recovery when versions cannot be read PR 50842
  • update code owners for the bringup tools PR 50857
  • trace manifest validation script PR 50000
  • Enable usage of dispatch engine in Quasar PR 48929
  • tools: cabling_generator: Improve hierarchical descriptor handling PR 50737
  • Many-to-many + regex/range patterns for MGD ASIC pinnings PR 51101
  • Add back device kernel time columns in perf report on quasar PR 51425
  • DIIM-278 - MPI stress test added to validation and recover PR 51317

Other

  • Feature: Add single-device fused moe_compute support PR 49886
  • requires_grid_size: enforce every marker, not just the closest PR 50334
  • Enable UnpackToDestFp32 in matmul when appropriate PR 50250
  • sweeps: exit run early on infrastructure failure (fabric bring-up or device-fatal run_mailbox wedge) PR 50694
  • Fix ttnn.all_gather Fabric_2D correctness bugs PR 51035
  • sweeps: honour skip_negative_entries in the plus_one golden PR 51299
  • Reject wrong combination for fused_rms_minimal op PR 51077
  • Support 1D tensors in all_reduce op PR 51347
  • Fix checks in all_gather PR 51239
  • Galaxy lead-model sweeps: fail fast on unrecoverable resets, cut device reopens, and stop reporting infra failures as test failures PR 51561
  • Move sparse_sdpa / indexer_score perf checks to the realtime device profiler PR 49878
  • Update Blackhole SDPA perf targets PR 50758
  • binary_ng: re-apply program-cache-hit args via override_runtime_arguments (fixes SDXL in-place PCC) PR 49828
  • Fix CIv2 LFC weight downloads (bypass proxy + retry on connection-refused) PR 49162
  • New ttnn.all_gather PR 48301
  • Add GitHub Agentic Workflows scaffolding and daily-repo-status workflow PR 50535
  • Move SDXL tests to Tier 2 CI PR 50006
  • 4x4 subtorus mesh support for CCLs PR 49881
  • remove deprecated global-namespace CoreCoord alias PR 50779
  • DeepSeek: Add mixed-format KV cache for sparse MLA PR 50207
  • trace manifest validation script PR 50000
  • Sweep infra: persistent per-job mesh device for Galaxy lead-models PR 50139
  • Blaze prefill (route perf jobs to high-power galaxy pool) PR 51187
  • mla: add 2D all-to-all for GLM sparse attention PR 49786
  • #45830: ring_mla — trace-safe per-element-tensor metadata overload (op + test, no trace) PR 48907
  • pre-commit: forbid ProgramDescriptor rebuilds inside override_runtime_arguments PR 51590
  • ttnn sweeps → pipeline-reorg: collapse per-HW caller jobs PR 50430
  • Ring reduce-scatter: contiguous intermediate PR 50933
  • fix install script on aarch64 PR 46042
  • Clarify container-based installation PR 45693
  • Add BH GLX Wan2.2 tests to CI PR 50939
  • Add GPT-OSS sliding prefill to RingJointSDPA PR 51438
Assets 27
Loading