Support CHI-C2C protocol modeling in gem5 Ruby CHI

[tsyw]

Hi all,

I have been extending gem5's Ruby CHI model to support coherent chip-to-chip communication for chiplet-style systems. The current proof-of-concept branch is dev_chi_c2c_poc. This is personal research work, not a corporate project.

The main references are ARM IHI 0098 Issue A.b and Fabian Schätzle's CHI C2C work. The implementation is not intended to be a complete bit-accurate model of the full C2C stack yet. The goal is to add enough protocol structure to gem5's existing Ruby CHI model to study coherent multi-chip systems, directory behavior, traffic distribution, and inter-chip bottlenecks.

I would like feedback on whether this direction is useful for upstream gem5, what subset would be appropriate for review, and where maintainers would prefer the boundary between reusable simulator support and project-specific evaluation infrastructure.

Why this matters

For CHI-based systems, a chip boundary is more than an added latency term. Once coherent traffic leaves a chip, the model needs to answer questions such as:

• where the boundary controller sits in the coherence topology,
• how request identity is preserved across gateway rewrites,
• how remote sharers are represented in the home directory,
• how snoops are routed when remote caches sit behind gateways, and
• how inter-chip bandwidth, queueing, and credit return affect progress.

gem5 already has a strong Ruby CHI foundation. Extending that foundation to multi-chip coherent systems would make it possible to study chiplet memory systems without replacing the coherence model with an abstract traffic-only approximation.

What exists in the code today

The implementation is centered on a separate MachineType:CHI_C2C Ruby controller rather than folding gateway behavior into an existing cache or home node. The controller has on-chip and C2C-side buffers for REQ, SNP, RSP, DAT, and MISC traffic.

The gateway currently models:

• request, response, data, and snoop forwarding across chip boundaries,
• read, write, dataless, atomic, DVM, retry/reissue, and WritePush paths,
• unordered write response handling, where DBIDResp and Comp may arrive in either order and CompDBIDResp represents both,
• multi-chip configurations with multiple C2C gateways per chip.

There is also an optional local ReadShared merge optimization before traffic enters the C2C path. This is not a CHI C2C protocol feature; it is a Ruby-side gateway optimization that can coalesce compatible local ReadShared requests behind one outbound C2C read and fan the completion back out locally.

The code also carries the identity and directory state needed for gatewayed coherence:

• CHI messages carry original source, original transaction, source gateway, and remote-destination metadata.
• Local response matching is scoped by (src, txnId, addr); inbound C2C matching is scoped by (peerGateway, txnId, addr); outbound C2C traffic uses gateway-assigned transaction IDs.
• Same-address requests from different local SrcIDs can remain outstanding at the C2C gateway at the same time. The gateway uses transaction keys rather than an address-only TBE key, so one active line address does not serialize all independent requestors.
• Local transaction keys are FIFO-matched so a reused local transaction tuple does not consume a later TBE incorrectly.
• Merged read followers are bound to the leader TBE so their CompAcks are still matched precisely.
• The home directory tracks remote sharers and the gateway that proxies them.
• Snoop destination fixup preserves actual remote sharers while routing through the correct C2C gateway.

On the transport side, the branch includes C2CPacketizerBridge and Format X container/packetizer support. The bridge still forwards Ruby message objects, but packetization is active in the runtime path to model container occupancy, TX queue pressure, latency, and delayed credit return. A per-message-class C2C credit manager tracks REQ, SNP, RSP, and DAT credits and exposes related stats.

Topology and configuration support includes two-chip and four-chip CHI C2C setups, R2/R3 ruby_mem_test stress configurations, SE workload configurations, and focused directed protocol-flow tests.

Current scope and non-goals

This is a protocol-functional and timing-oriented CHI C2C model. It should be read as usable support for coherent multi-chip research in gem5, not as full IHI0098 compliance.

Current non-goals include:

• bit-accurate link-header semantics and serialized wire decode,
• a complete interface-management/property-exchange handshake,
• a strict reproduction of the spec's DN/MN decomposition for every DVM path,
• RME-related support, and
• upstreaming the full experiment and plotting pipeline as part of the initial simulator feature.

Open issues

LCN support is an open design issue. In CHI C2C(actually, this is a CMN function), LCN is an HN-side function that allows a local domain to cache remote addresses; conceptually it behaves like an extra cache level between the Remote HN and the Local RN. The current branch models C2C gateway traversal, remote-sharer tracking, snoop routing, packetization, and credit timing, but it does not yet model an LCN as an HN-side cache for remote-home lines. I would like maintainer feedback on whether LCN should be modeled as an extension of the CHI home-node path, as a separate cache-like Ruby controller adjacent to the C2C gateway, or as follow-on work after the basic gateway and transport support are upstreamed.

Validation coverage

The current validation set covers the main protocol and transport surfaces:

Coverage	What it checks	Command or entry point
CHI C2C regression suite	Registered integration tests for directed flows, R2/R3 stress, and atomic-forwarding cases.	cd tests && ./main.py run tests/gem5/chi_c2c
Directed protocol flows	SeriesGetx, SeriesGets, SeriesGetMixed, and Invalidate traffic through a two-chip topology with packetizer bridges and deferred credit returns.	./build/ARM/gem5.opt tests/gem5/chi_c2c/configs/chi_c2c_run.py --test-type SeriesGetMixed --requests 100 --topology=Crossbar
Larger directed stress	Same directed generators with more requestors or larger request windows, useful for queueing, retry, and ordering pressure.	./build/ARM/gem5.opt tests/gem5/chi_c2c/configs/chi_c2c_run.py --test-type Invalidate --requests 100 --num-cpus 4 --topology=Crossbar
Local read-merge optimization	Gateway-side ReadShared merging before traffic enters the C2C path; this is local Ruby optimization coverage, not CHI C2C protocol coverage.	./build/ARM/gem5.opt tests/gem5/chi_c2c/configs/chi_c2c_run.py --test-type SeriesGetMixed --requests 100 --enable-read-merge
R2 ruby_mem_test stress	Multi-requestor random traffic through an R2-style CHI C2C topology.	./build/ARM/gem5.opt tests/gem5/chi_c2c/configs/chi_c2c_ruby_mem_test.py --topology R2 --num-cpus 4 --abs-max-tick 10000000000
R3 ruby_mem_test stress	Three-gateway random traffic through an R3-style CHI C2C topology.	./build/ARM/gem5.opt tests/gem5/chi_c2c/configs/chi_c2c_ruby_mem_test.py --topology R3 --num-cpus 4 --num-c2cgs 3 --abs-max-tick 10000000000
Atomic forwarding	Atomic-only R2/R3 traffic to exercise remote forwarding and completion paths.	./build/ARM/gem5.opt tests/gem5/chi_c2c/configs/chi_c2c_ruby_mem_test.py --topology R3 --num-cpus 4 --num-c2cgs 3 --atomic 100 --abs-max-tick 1000000
Packetizer/container unit tests	Format X container construction and packetizer behavior.	scons build/ARM/mem/ruby/protocol/chi/c2c/C2CPacketizer.test.opt && ./build/ARM/mem/ruby/protocol/chi/c2c/C2CPacketizer.test.opt
Credit accounting	Runtime C2C REQ/SNP/RSP/DAT credit consumption and delayed return, covered by packetized directed and ruby_mem_test runs.	./build/ARM/gem5.opt tests/gem5/chi_c2c/configs/chi_c2c_run.py --test-type SeriesGetMixed --requests 100
SE workload configurations	Representative syscall-emulation workloads used for performance studies rather than minimal protocol regressions.	./build/ARM/gem5.opt tests/gem5/chi_c2c/configs/chi_c2c_se.py <workload arguments>

That does not mean the work is finished, but it does mean the discussion can be about upstream scope, interfaces, and review shape rather than only raw feasibility.

What seems upstreamable first

My current view is that this should be split into a few reviewable layers.

1. Core CHI C2C modeling support

• the CHI_C2C controller,
• CHI message metadata needed for gateway traversal,
• remote-sharer tracking in the directory path, and
• configuration hooks for multi-chip CHI topologies.

2. Transport and packetization support

• C2CPacketizerBridge,
• Format X container/packetizer support as currently modeled,
• bridge-side queueing and delayed credit return, and
• focused tests for packetization and credit accounting.

3. Optional follow-on work

• LCN modeling for remote-home lines,
• richer initialization and property-handshake support,
• broader spec coverage,
• full-system polish for additional topologies, and
• experiment-specific workload and plotting infrastructure.

My bias would be to discuss layers 1 and 2 as the initial upstream target and treat layer 3 as follow-on work unless maintainers want a broader first series.

Questions for maintainers and CHI modelers

I would especially value feedback on the following:

1. Is explicit CHI C2C support in Ruby useful for upstream gem5?
2. Is a separate CHI_C2C controller the right architectural boundary?
3. Is the current transport scope acceptable for an initial contribution: runtime Format X packetization, bridge latency/TXQ modeling, and delayed credit return, while leaving full link-header semantics out of scope?
4. Is the remote-sharer to gateway mapping in the home directory a reasonable abstraction?
5. How should LCN be represented if it is added later: inside the home-node path, as a separate cache-like Ruby controller, or behind another abstraction?
6. What validation bar would be expected for upstream review: focused unit tests plus protocol regressions, or broader end-to-end workload evidence as well?
7. Would maintainers prefer a staged patch series that lands the controller, transport model, and topology/configuration support separately?

If there is interest, the next step would be to break this into a concrete patch plan with a minimal initial subset, explicit non-goals, and tests that make that subset reviewable.

[tsyw]

Updated for LCN disscussion and add a table for the tests used in my PoC branch.