Cyclic dependency finder based on cmssw source (with or with header only dependencies)

[davidlange6]

usage: cmssw_cycle_finder.py [-h] [--ignore_header_only] [--status_bar]

CMSSW Cyclic dependency finder

optional arguments:
-h, --help show this help message and exit
--ignore_header_only Ignore cycles due to header only dependencies
--status_bar Show progress bar when running

Requires python3.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31425/18286

• This PR adds an extra 12KB to repository

[cmsbuild]

A new Pull Request was created by @davidlange6 (David Lange) for master.

It involves the following packages:

Utilities/ReleaseScripts

@makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please review it and eventually sign? Thanks.
@wddgit this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31425/18287

• This PR adds an extra 16KB to repository

[davidlange6]

please test

[cmsbuild]

The tests are being triggered in jenkins.

• CMSSW_11_2_X_2020-09-09-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/9248/console Started: 2020/09/10 14:08

[cmsbuild]

Pull request #31425 was updated. @makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please check and sign again.

[davidlange6]

output on CMSSW_11_2_X_2020-09-08-2300

Building adjacency graph
Looking for cycles
Cycle detected: found a back edge from DataFormats/TrackerCommon to DataFormats/SiPixelDetId. It involves
DataFormats/TrackReco, DataFormats/SiPixelDetId, Alignment/CommonAlignment, TrackingTools/TrajectoryState, DataFormats/TrackerCommon, TrackingTools/TransientTrackingRecHit
Cycle detected: found a back edge from SimCalorimetry/HGCalSimProducers to SimCalorimetry/HGCalSimAlgos. It involves
SimCalorimetry/HGCalSimAlgos, SimCalorimetry/HGCalSimProducers

[cmsbuild]

+1
Tested at: 1829ddb
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5e3961/9248/summary.html
CMSSW: CMSSW_11_2_X_2020-09-09-2300
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31425/18309

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #31425 was updated. @makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please check and sign again.

[smuzaffar]

@davidlange6 , if you are only doing it using the CMSSW_RELEASE_BASE then why not make use of already created dependency information $CMSSW_RELEASE_BASE/etc/dependencies/uses.out.gz? Each line in this file is

src_file space_separated_list_of_all_included_headers

[davidlange6]

remind me how that list gets made? (I had forgotten about it even if I probably asked for it or helped create it..)

[smuzaffar]

We run this https://github.com/cms-sw/cmssw-config/blob/scramv3/SCRAM/findDependencies.py script for every release which basically parses the .d files generated by compiler. So one drawback/feature of this is that it only contains those files which are actually compiled and included. For example any header file which is not included by any source file will not end up in this.

[davidlange6]

Thanks! Then indeed this can replace what I’ve done completely when the option of omitting header only dependencies is selected. On Sep 11, 2020, at 6:03 PM, Malik Shahzad Muzaffar <notifications@github.com<mailto:notifications@github.com>> wrote: We run this https://github.com/cms-sw/cmssw-config/blob/scramv3/SCRAM/findDependencies.py script for every release which basically parses the .d files generated by compiler. So one drawback/feature of this is that it only contains those files which are actually compiled and included. For example any header file which is not included by any source file will not end up in this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#31425 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGPFQ66ARXMUZEAIJ2IP5LSFJC3TANCNFSM4RE33NMQ>.

[silviodonato]

@davidlange6 if I understand correctly, we close this PR. Is it correct?

[smuzaffar]

please test
@silviodonato , this is still a useful helper script. I would say lets get this in once tests passed (basically python3 test)

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5e3961/16060/summary.html
COMMIT: 74f39a5
CMSSW: CMSSW_12_0_X_2021-06-16-0900/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/31425/16060/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2862520
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2862497
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
• Checked 160 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found

[smuzaffar]

+core

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1