HPM Questions

[abejgonzalez]

Moving the question from #152 about HPMs to here

Adapted from @heiner-bauer s question:

After reading chapter 12 of the documentation I have several questions regarding HPMs / HPEs:

• Which events does BOOM track? The number and ordering of the counters in BOOM does not seem to match the SiFive U54 events.
• How many independent counters does BOOM support?
• Do you happen to have some kind of library to use these counters (configuring and reading these CSRs is kind of fiddly)?

[abejgonzalez]

Thanks for the question!

First of all, any suggestions to clarify the documentation is always helpful. Sorry it didn't help to much in this case.

1. Which events does BOOM track?

Here are the snippets of code that define the events.

riscv-boom/src/main/scala/exu/core.scala

Lines 190 to 195 in f4968d7

	val perfEvents = new freechips.rocketchip.rocket.EventSets(Seq(
	new freechips.rocketchip.rocket.EventSet((mask, hits) => (mask & hits).orR, Seq(
	("exception", () => rob.io.com_xcpt.valid),
	("nop", () => false.B),
	("nop", () => false.B),
	("nop", () => false.B))),

riscv-boom/src/main/scala/exu/core.scala

Lines 221 to 228 in f4968d7

	new freechips.rocketchip.rocket.EventSet((mask, hits) => (mask & hits).orR, Seq(
	("I$ blocked", () => icache_blocked),
	("nop", () => false.B),
	("branch misprediction", () => br_unit.brinfo.mispredict),
	("control-flow target misprediction", () => br_unit.brinfo.mispredict &&
	br_unit.brinfo.is_jr),
	("flush", () => rob.io.flush.valid),
	("branch resolved", () => br_unit.brinfo.valid))),

riscv-boom/src/main/scala/exu/core.scala

Lines 245 to 251 in f4968d7

	new freechips.rocketchip.rocket.EventSet((mask, hits) => (mask & hits).orR, Seq(
	("I$ miss", () => io.ifu.perf.acquire),
	("D$ miss", () => io.dmem.perf.acquire),
	("D$ release", () => io.dmem.perf.release),
	("ITLB miss", () => io.ifu.perf.tlbMiss),
	("DTLB miss", () => io.dmem.perf.tlbMiss),
	("L2 TLB miss", () => io.ptw.perf.l2miss)))))

They tracks things such as branch prediction, ROB flushes, I/D$ misses, etc. As for the numbering/ordering of the events, we reference the U54 manual as an example on how to access events (if I remember correctly they map to the Rocket code). The events that they have (and what Rocket has) will be different from BOOM. You need to look at the exu/core.scala to see the signals that we chose and access them using the EventSet and EventBit that is relevant.

Note: If you want to track your own event, feel free to modify the source and add your own.

2. How many counters do we support?

We use the same infrastructure to support counters as Rocket. So if I am not mistaken we support 32 (including cycle, time, instret counters).

3. A library to use the counters (configuring and reading them is kinda ugly)?

What I end up using for my own projects is this repository: https://github.com/abejgonzalez/riscv-code-constructor. It includes a header file https://github.com/abejgonzalez/riscv-code-constructor/blob/master/inc/hpm-util.h and example on how to use it https://github.com/abejgonzalez/riscv-code-constructor/blob/master/src/hpmCounters.c. Otherwise, riscv-tests uses setStats to read the counters.

https://github.com/riscv/riscv-tests/blob/fbf5f3a2589c61d34569524dbf353beda0b6b4de/benchmarks/common/syscalls.c#L40-L54

Finally, Chris Celio created a program to help reading counters also: https://github.com/ccelio/riscv-hpmcounters

Otherwise, as far as I know, there is no "de-facto" standard for reading out counters. So you have to hack something together or use the code found in one of these sources (or somewhere else that I don't know of).

Please let me know if you have any other questions! I can also update the docs with this information if that was unclear.

[heiner-bauer]

Thanks for the pointers to software examples.

So the mapping is like this ?

mhpeventX[7:0] = 0

Bits	Meaning
8	Exception

mhpeventX[7:0] = 1

Bits	Meaning
8	I$ blocked
9	-
10	Branch misprediction
11	Control-flow misprediction
12	Flush
13	Branch resolved

mhpeventX[7:0] = 2

Bits	Meaning
8	I$ miss
9	D$ miss
10	D$ release
11	ITLB miss
12	DTLB miss
13	L2 TLB miss

Could you please describe what exactly "Flush" and "I$ blocked" means ?

I assume that at unspecified bit positions( e.g. with mhpeventX[7:0] = 1 at position 14) nothing happens, i.e. the counters will not increment.
Why is it then necessary to have "nop", () => false.B ?

Regarding the number of counters, I just stumbled over nPerfCounters in Chapter 17, maybe this should be mentioned together with a more detailed description of the supported HPEs / HPMs ?

[abejgonzalez]

Yup those mappings look right to me.

Flush goes high whenever the ROB is flushed due to things such as exceptions, pipeline replays, and mem. ordering failures. Its used to redirect the frontend of the pipeline through the Fetch Target Queue. For reference:

riscv-boom/src/main/scala/ifu/fetch-target-queue.scala

Lines 174 to 194 in 317b137

	when (io.deq.valid || io.flush.valid) {
	assert (!(io.deq.valid && io.flush.valid && io.deq.bits =/= io.flush.bits.ftq_idx),
	"FTQ received conflicting flush and deq on same cycle")
	commit_ptr := Mux(io.flush.valid, io.flush.bits.ftq_idx, io.deq.bits)
	}
	//-------------------------------------------------------------
	// **** Handle Mispredictions/Flush ****
	//-------------------------------------------------------------
	when ((io.brinfo.valid && io.brinfo.mispredict) || io.flush.valid) {
	// Flush signal is sent out at commit, so that should override
	// earlier branch mispredict signals.
	val new_ptr = WrapInc(Mux(io.flush.valid,
	io.flush.bits.ftq_idx,
	io.brinfo.ftq_idx),
	num_entries)
	enq_ptr.value := new_ptr
	// If ptr is adjusted, we deleted entries and thus can't be full.
	maybe_full := (enq_ptr.value === new_ptr)
	}

I$ blocked:

riscv-boom/src/main/scala/exu/core.scala

Line 258 in 317b137

val icache_blocked = !(io.ifu.fetchpacket.valid || RegNext(io.ifu.fetchpacket.valid))

It checks if there is a valid Fetch Packet coming from the Frontend.

As for the nop signals, with false.B they will not get incremented, correct. Why are they there? I don't exactly know, my guess is that they were put to original space out the BOOM HPMs so they matched Rocket, but over time both cores diverged and these were not updated. But as I mentioned earlier, feel free to override and change these (or if you find some useful signals to profile, put in a PR for the world to see!)

My main hesitation about adding a specific list of HPM/E's that BOOM support is that it could cause future divergence if the docs are not up to date. That's why I just point to the exu/core.scala file. However, I think a valid thing is to mention the number of counters we support.

[abejgonzalez]

@heiner-bauer Any other questions on HPM's or is this issue resolved?

[heiner-bauer]

Yes, you can close this issue.
Thanks for your help !

[abejgonzalez]

Great. Let us know if you have any more questions!