profiling verilator

[agrobman]

@wsnyder ,

using 4.200,

command:

verilator  -cc -CFLAGS "-std=c++11 -g -pg" --prof-cfuncs  -Wno-WIDTH -Wno-UNOPTFLAT -Wno-IMPLICIT -Wno-MULTIDRIVEN --autoflush   -f flist -v gitstate.v +define+USE_GIT_STATE --top-module tb_top -exe test_tb_top.cpp
make -j -C obj_dir/ -f Vtb_top.mk OPT_FAST="-Os"

I didn't get promised gmon.out (?) after simulation run ..

What do I miss?

[agrobman]

the individual files are compiled as:

ccache g++ -I. -MMD -I../verilator/4.200/include -I/../verilator/4.200/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=0 -DVM_TRACE_FST=0 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow -std=c++11 -g -pg -std=gnu++14 -Os -c -o SimJTAG.o ../testbench/SimJTAG.cc

[wsnyder]

GCC also requires you link with profiling, so also add -LDFLAGS "-pg"

[agrobman]

Thanks a lot

Verilator doc does not mention LDFLAGS - probably UM needs to be fixed about this, also I had to run other programs as:

gprof <verilator executable> gmon.out > prof.txt
verilator_profcfunc prof.txt

which was not clear from UM ..

[wsnyder]

Fair point - could you please make a quick pull request to add those steps to the docs?

[agrobman]

I'm not sure how to do this, it will be better if somebody, more familiar to the process, does this ..

(also will be nicer if the verilator_profcfunc will sort output for modules based on % of time ...)

For our design it looks like:( sorted by module name (?))

Overall summary by module:
  % time  module
    0.44  C++
    0.04  DW01_addsub
    0.08  DW02_multp
    0.48  DW_lsd
    0.77  DW_lzd
    0.05  DW_minmax
    0.02  SimJTAG
    5.84  VLib
   26.53  Vtb_top common code
   22.08  beh_lib
    0.02  core_events_counters
    0.01  dmi_jtag_to_core_sync
    0.09  ehx3_biu
    0.14  ehx3_dbg
    0.02  ehx3_dec
    0.08  ehx3_dec_csr
    4.25  ehx3_dec_decode_ctl
    0.56  ehx3_dec_frf64_ctl
    0.66  ehx3_dec_gpr_ctl
    1.42  ehx3_dec_ib_ctl

[agrobman]

I would think the --prof-* option should be sufficient to set up all other needed compilation flags inside the generated makefile and also to run report generated tools after simulation ...

[wsnyder]

Fair point about adding -pg automatically. I pushed doing this. You'll still need to run gprof manually as Verilator never runs a simulation itself, so it obviously can't do anything "after" the simulation either.

[agrobman]

Still verilator can enable code in executable to run post-processing tools at the exit from simulation , based on the --prof switch.

[wsnyder]

The executable can't do it as the data isn't written until exit() is called.

[wsnyder]

Also the user currently owns main(), so this is outside the scope of what I'd like Verilator to do - for now anyways.

[agrobman]

can't the "executable file" be a script, running real binary and then postprocessing tools?

[wsnyder]

It could be, but the executable and main() is owned by the user at present, and adding a wrapper to change that is a very significant step and effort. This does not seem worth it (vs investing in fixing other items) just to further simplify profiling which is not a common use case.

[futurehome]

@wsnyder Does gprof has limitation on design scales? I found there is no gmon.out for my whole design, but there is gmon.out if I switch to one of the submodules.

[wsnyder]

gprof works on C programs, no concept of designs. If you dont get it at some levels you are not using -pg at all complie and links, or are not exiting the simulation cleanly ($finish or any path to C exit()).

[futurehome]

I use "--prof-cfuncs -LDFLAGS '-pg' CFLAGS '-pg' " in verilator command line, and I'm sure all the makefile generated by verilator has -pg for both compiling and link.
I did another test, with one large module, which instantiate a sub-module multiple times, generate cmodel in flat mode, and I found that it coundn't generate gmon.out for the complete design(about 2025 cpp files), but if I reduce the times of instantiation to some small number(about 1203 cpp files), it can generate gmon.out. It looks like it has some relation with the size of the c-code?

[wsnyder]

it shouldn't, I've never seen a problem myself. There is also as a -pg alternative, oprofile and many other C program profile tools you could use. This profiling is all standard C profiling, unrelated to Verilator, so search around for general C tracing advice.

[futurehome]

Looks like -pg has some issue with large design.
If using oprofile, is it still possible to use verilator_profcfunc to analyze the data?

[wsnyder]

gcc 3 is ancient, I wouldn't trust that. But I haven't tried feeding in oprofile data, if a change is needed to parse it, a pull would be welcome.

[futurehome]

I've confirmed that because one of the array in gprof is out of range due to too many functions in cpp, it can't dump gmon.out. I have to switch to perf or gperftools for profiling.

[wsnyder]

This page https://www.gnu.org/software/libc/manual/html_node/gmon-Tunables.html

says change the glibc.mem.maxarcs and glibc.mem.minarcs tunables to fix this. Seems default is MAXARCH in gmon.h of 1<<20, so e.g. try

export GLIBC_TUNABLES=glibc.mem.maxarcs=10000000:glibc.mem.minarcs tunables=10000000