Skip to content

Add multi-group support for CachePool - #27

Merged
Aquaticfuller merged 11 commits into
mainfrom
dev/multi-group-rebase
Aug 14, 2026
Merged

Add multi-group support for CachePool#27
Aquaticfuller merged 11 commits into
mainfrom
dev/multi-group-rebase

Conversation

@DiyouS

@DiyouS DiyouS commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Extends CachePool from a single group of tiles to a configurable 2D mesh of groups, each connected via a FlooNoC-based inter-group
interconnect, plus a matching L2 refill mesh and a hardware partial-barrier mechanism scoped to individual tiles.

Multi-group cluster architecture

  • New hierarchy level: Cluster → Group × N (FlooNoC 2D mesh) → Tile × M → Core Complex + L1 cache. Groups are arranged in an X×Y mesh
    (num_groups_x/num_groups_y), each exposing per-tile NoC ports ("planes") that connect directly to the corresponding plane on
    X/Y-neighboring groups.
  • Tile IDs are globally unique and flat (tile_id = tile_base_id + t = (gy*NumGroupsX + gx) * NumTilesPerGroup + t), so a tile's group can
    be recovered directly from the upper bits of its ID.
  • cachepool_group_noc_wrapper handles the inter-group boundary per group: a per-tile concentration crossbar, a floo_router pair (request +
    response) per tile/plane, and a group-wide dispatch crossbar back into the tile's local remote ports.
  • Each tile's TCDM crossbar (tcdm_cache_interco) now does 3-way routing per request: local cache bank, intra-group remote (same group,
    different tile), or inter-group remote (different group) — selectable per-config via num_rg_ports_per_core; setting it to 0 disables all
    multi-group RTL and falls back to single-group behavior.
  • BootROM and the refill crossbar moved to group level to scale with group count.

L2 refill NoC

  • Replaces the previous point-to-point L2/DRAM refill path with a TCDM-based mesh spanning groups, so every group can reach every L2/HBM
    channel regardless of physical placement in the mesh.
  • Per-group reqrsp_to_axi conversion happens before the mesh hop, with a group-count-aware AXI mux stage merging traffic onto the shared
    wide AXI ports.

Partial barrier (tile-level, direct-wire)

  • New direct-wire cluster barrier (cachepool_cluster_barrier) that collects per-tile barrier requests and broadcasts completion, replacing
    a full-cluster-only barrier with one that supports excluding tiles via a runtime-configurable participation mask.
  • Participation mask is software-writable through a new peripheral CSR (HW_BARRIER_PARTICIPATION_MASK, multi-word to scale past 32 tiles)
    and exposed to the runtime via snrt_barrier APIs, so a kernel can synchronize a subset of tiles instead of the whole cluster.
  • New partial_barrier and partial_barrier_benchmark test kernels; the existing multi-producer/consumer RLC kernel was adapted to use it.

NoC visualization / profiling

  • Cycle-accurate per-router/tile/core NoC traffic logging (hardware/tb/cachepool_noc_profiling.sv), opt-in via noc_profiling=1, consumed
    by util/scripts/noc_profiling_to_vis4mesh.py to render mesh traffic heatmaps (see README's "NoC Visualization" section).

New configs

  • cachepool_fpu_4g (2×2 mesh), cachepool_fpu_16g / cachepool_fpu_16g_tiny (4×4 mesh) alongside the existing single-group configs.

TODOs

  • Pass lint checks (Ongoing)
  • Fix a bug in rsp fifo handling
  • Fix the waveform script for cache controller
  • Add new scoreboard system in interconnection for debugging

DiyouS added 10 commits August 11, 2026 18:27
Move BootROM and the refill xbar to group level for future scaling,
add multi-group support (cross-group interconnection WIP), add FlooNoC
as a dependency, and default to an all-private cache config for early
multi-group bring-up/CI.
Wire up the inter-group NoC configuration and connections, add the L2
ICache, plus assorted cleanups, renames, and cache init/flush fixes
along the way.
Bump Spatz and Bender dependencies; update fence implementation and add
DRAM allocation functions in the runtime; move DRAM type settings into
config files with auto-calculated refill data width; config/CI cleanup
in preparation for the L2 Refill NoC rework.
Switch to a TCDM-based mesh for the L2 refill path (major update, plus
follow-up performance and functional fixes), add a 4x4 config for
faster testing, remove the no-longer-working 2g configs, and assorted
CI/DEBUG flag and loadstore-test fixes.
Add partial-barrier support to the cluster using the current direct-wire
barrier implementation, adjust the RLC kernel to use it, and rename some
configuration/hardware parameters for readability.
Add a NoC profiling/monitoring module and a Makefile target to generate
the corresponding vis4mesh dataset for performance analysis.
Fix the gemv kernel's result store buffer issue, add hardware cuts for
peripheral signals for physical design, and fix an L1 NoC bug that
caused responses to always be sent on a single channel.
Replace the .gitlab-ci.yml PATH override and make quick-tool symlink with
iis-env.sh, which exports INSTALL_DIR for the prebuilt toolchain and sets
up a Python venv (with pip upgrade and numpy) for code-gen dependencies,
matching the new clean-runner CI environment.

[Util] Update vis4mesh data generation.
@DiyouS
DiyouS requested a review from Aquaticfuller August 12, 2026 09:35
@DiyouS DiyouS self-assigned this Aug 12, 2026
@DiyouS DiyouS added feature New feature to the design major RTL labels Aug 12, 2026
- Update insitu-cache dependency for a bug fix
- Remove debugging scoreboard in interconnect (broke multi-tile config)
- Fix Spatz store counter bug
- Update run_all.sh to avoid log folder missing issue
- Fix barrier bug caused by added cuts
@DiyouS
DiyouS force-pushed the dev/multi-group-rebase branch from a56c7a4 to 3ec4e52 Compare August 13, 2026 14:54
@Aquaticfuller
Aquaticfuller merged commit 044e768 into main Aug 14, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature to the design major RTL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants