No execuation time comparison available for PRs

[mandrenguyen]

Since a few months we are not able to see the CPU impact of a given pull request, which used to be possible with the enable profiling option in the Jenkins tests.
This is a bit problematic for integrating new features, as we won't easily be able to keep track of changes in performance until a pre-release is built.
The issue seems to come from igprof, which apparently can no longer really be supported.
One suggestion from @gartung is to try to move to VTune.

[mandrenguyen]

assign core, reconstruction

[cmsbuild]

New categories assigned: core,reconstruction

@Dr15Jones,@jfernan2,@makortel,@mandrenguyen,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @mandrenguyen Matthew Nguyen.

@rappoccio, @antoniovilela, @sextonkennedy, @makortel, @smuzaffar, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[gartung]

The problem is that the cmsRun process itself gets a segfault while being profiled by Igprof. The same segfault might happen when being profiled with Vtune.

[makortel]

In case IgProf+cmsRun combination crashes, is any information on the job timings saved that can be used for comparison?

[gartung]

Usually the FastTimerService job completes and the average per module is contained in the raw json file if the resources piechart is not readable.

[gartung]

The IgprofService dumps the profile after the first, middle and next to last event. The first one might not have enough data to be meaningful.

[gartung]

@mandrenguyen Can you point me to a PR so I can look at the logs.

[mandrenguyen]

@gartung The last one we tried with profiling enabled was:
#43107

[makortel]

The crashes under profilers are quite likely caused by the memory corruption inside Tensorflow (when ran through IgProf or VTune) that has been investigated in #42444.

[VinInn]

The FastTimer Service should suffice. Still It seems not active in RelVals