Alignment/MillePedeAlignmentAlgorithm/test_CreateFileLists: give a better error message in case of failure

[iarspider]

PR description:

test_CreateFileLists failed in CMSSW_12_5_ASAN_X_2022-06-06-1100 IB: log

Traceback (most recent call last):
  File "/cvmfs/cms-ib.cern.ch/week0/el8_amd64_gcc10/cms/cmssw/CMSSW_12_5_ASAN_X_2022-06-06-1100/bin/el8_amd64_gcc10/mps_create_file_lists.py", line 12, in <module>
    exec(open(subprocess.check_output(["which", "tkal_create_file_lists.py"], universal_newlines=True).rstrip()).read())
  File "<string>", line 1174, in <module>
  File "<string>", line 44, in main
  File "<string>", line 120, in create
  File "<string>", line 375, in _request_dataset_information
  File "/cvmfs/cms-ib.cern.ch/week0/el8_amd64_gcc10/external/python3/3.9.6-67e5cf5b4952101922f1d4c8474baa39/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
UnboundLocalError: local variable 'das_data' referenced before assignment

This PR fixes the error by initializing das_data with some generic state before running DAS query. Also, the PR updates test definition to use tkal_create_file_lists.py instead of mps_create_file_lists.py as indicated in mps_create_file_lists.py:

=======================================
This script is moved to its new home in
tkal_create_file_lists.py              
(in Alignment/CommonAlignment/scripts).
=======================================

PR validation:

Test ran locally without errors

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38271/30422

• This PR adds an extra 24KB to repository

[cmsbuild]

A new Pull Request was created by @iarspider for master.

It involves the following packages:

• Alignment/CommonAlignment (alca)
• Alignment/MillePedeAlignmentAlgorithm (alca)

@cmsbuild, @malbouis, @tvami, @yuanchao, @francescobrivio can you please review it and eventually sign? Thanks.
@pakhotin, @mmusich, @adewit, @tocheng, @tlampen this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[iarspider]

please test

[tvami]

@cmsbuild , please test for CMSSW_12_5_ASAN_X

[tvami]

type bugfix,trk

[smuzaffar]

shouldn't we move this unit tests to Alignment/CommonAlignment as tkal_create_file_lists.py exists in that package. Any change in this script will then allow its automatic testing

[smuzaffar]

I think this is not going to fix the unit test but it should give a proper error message. I think test randomly fails for ASAN IBs e.g. kibana shows that it works some time.

[iarspider]

I think this is not going to fix the unit test but it should give a proper error message.

Exactly. I will update the PR title.

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-66f74b/25342/summary.html
COMMIT: 60f2be1
CMSSW: CMSSW_12_5_ASAN_X_2022-06-06-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38271/25342/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @bsunanda Phase2-hgx315 Modify the cfg's in Validation/HGCalValidation to accommodate V17 version of HGCal geometry #38240
• @cms-trackeralign Introduce TkAlJetHT #38254
• @sroychow Pixel Heterogeneous DQM: add 1D differnce plots in vertex comparison module. #38245
• @quark2 Fix on issue #38044 #38101
• @grasph Bug fix for the EcalDumpRaw expert tool, which is not used in the standard workflow. #38176
• @ChrisMisan Fix Sampic DQM channel_per_plane histograms #38210
• @ChrisMisan Fix for hardcoded subdetector id in PPS reconstruction code #38208
• @smuzaffar [GCC12] Fix address of object will never be NULL error #38263
• @jeongeun Cleanup redundant edm ParameterSet exist in RecoLuminosity #38218
• @Dr15Jones Fix CMS deprecation warning in L1Trigger/TextToDigi #38267
• @srimanob Remove HLT process name out from L1CaloTrigger #37988

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-66f74b/25342/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-66f74b/25342/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test test_MpsWorkFlow had ERRORS

[tvami]

I understand that this #38271 (comment) is expected. Shall this change #38271 (comment) be done before approving?

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-66f74b/25339/summary.html
COMMIT: 60f2be1
CMSSW: CMSSW_12_5_X_2022-06-07-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38271/25339/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3652078
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3652054
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[smuzaffar]

I understand that this #38271 (comment) is expected.

yes

Shall this change #38271 (comment) be done before approving?

yes, this change be done in a separate PR

[tvami]

+alca

• see Alignment/MillePedeAlignmentAlgorithm/test_CreateFileLists: give a better error message in case of failure #38271 (comment)

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1