MTD geometry: update etlv5 setup to fit in D75-D76 envelope without overlaps, move unit tests to D76

[fabiocos]

PR description:

In this PR the MTD geometry scenario I13 (with ETL v5), as recently added to scenario D75, is updated (by @martatornago) to fit without overlaps in the external envelope available. Some ad hoc z shifts are removed, and the x-y order of the module layout is reviewed for a consistent treatment fo both disc faces. x-y sensor positions are unchanged overall, but the assignment of ETLDetId changes.

The scenario D75 is now used as the baseline for unit tests in both the Geometry and the RecOMTD/DetLayers packages. A corresponding update of the references used by Geometry unit tests is proposed in cms-data/Geometry-TestReference#5 , they need to be integrated at the same time in order to avoid failures.

PR validation:

Unit tests run, and a visual inspection shows the expected results. The upgrade test workflow 34234.0 for scenario D75 runs smoothly. The Geant4 overlap test does not show any residual issue in EndcapTimingLayer.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32840/21052

• This PR adds an extra 24KB to repository

[cmsbuild]

A new Pull Request was created by @fabiocos (Fabio Cossutti) for master.

It involves the following packages:

Geometry/MTDCommonData
Geometry/MTDGeometryBuilder
Geometry/MTDNumberingBuilder
RecoMTD/DetLayers

@perrotta, @civanch, @Dr15Jones, @makortel, @cvuosalo, @ianna, @mdhildreth, @cmsbuild, @srimanob, @kpedro88, @slava77, @jpata can you please review it and eventually sign? Thanks.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[fabiocos]

please test with cms-data/Geometry-TestReference#5

[fabiocos]

data packages cannot be directly tested by the bot without an update of cmsdist, hence the failure of Geometry/MTD* unit tests, as it should happen if the references are not properly updated

[cmsbuild]

-1

Failed Tests: UnitTests RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-199547/12757/summary.html
COMMIT: b6abe66
CMSSW: CMSSW_11_3_X_2021-02-07-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32840/12757/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test GeometryMTDGeometryBuilderTestDriver had ERRORS
---> test GeometryMTDNumberingBuilderTestDriver had ERRORS
---> test GeometryMTDCommonDataTestDriver had ERRORS

RelVals-INPUT

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 2963 differences found in the comparisons
• DQMHistoTests: Total files compared: 37
• DQMHistoTests: Total histograms compared: 2751765
• DQMHistoTests: Total failures: 74871
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2676872
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
• Checked 156 log files, 37 edm output root files, 37 DQM output files

[cvuosalo]

+1

[srimanob]

@fabiocos
Do I understand correctly that the test will keep false, but we should integrate it? I assume that this PR should be merged, so that we can do D75, D76 relvals properly.

By the way, why don't we use D76 as a baseline for unit test?

[fabiocos]

@srimanob the failure in unit test is simply due to the fact that the output of the test (D75 with revised etlv5) is compared with the reference from D50, I have provided the new references in cms-data/Geometry-TestReference#5 but they cannot be trivially used as done in my command, the need to be imported in a cmsdist update (@smuzaffar do you confirm)? So this PR should be updated in parallel with cms-data and cmsdist, otherwise you will get annoying failures in any IB and/or PR test just because of this.

The failure in comparison seems to have nothing to do with my PR (no MTD in 2021 geometry, I hope...). I am not familiar with this new matrix input test check, but it looks to be again unrelated to my updates.

[fabiocos]

@srimanob concerning the new baseline to be used, up to you to tell. From the MTD point of view D75 and D76 should be identical, if you prefer D76 I can easily modify this PR. I expect that soon we will want to use D77, but presumably it is too early. In any case, as I mentioned to @silviodonato , it would be good to have a test wf in the short matrix for the new baseline asap, so as to have a direct monitoring of it in any possible check. Please let me know

[smuzaffar]

@fabiocos , bot can test data packages (without cmsdist update). The data files from cms-data/Geometry-TestReference#5 should be available in the bot PR tests

[fabiocos]

@smuzaffar thanks for recalling me, so let me check once more the test failure. The cmsdist will be done at the integration time, right?

[fabiocos]

@srimanob ok, I understand the issue, I did not check carefully enough the test output: the unit test looks successful in the DDD part, while it fails in the DD4hep part because I only partly adapted the xml reference name. I am going to fix it, please confirm that you prefer D76 as new baseline.

[silviodonato]

please test 34234.0 with cms-data/Geometry-TestReference#5

[fabiocos]

@silviodonato as far as I can see the available test for D76 is 34634.0

17:11 farmui02 938> runTheMatrix.py -n | grep D76
34634.0 2026D76+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal 

, anyway D75 and D76 are identical as far as MTD is concerned

[fabiocos]

@silviodonato in my local test 34634.0 runs smoothly

[fabiocos]

please test workflow 34234.0,34634.0 with cms-data/Geometry-TestReference#5

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-199547/12804/summary.html
COMMIT: def38e4
CMSSW: CMSSW_11_3_X_2021-02-09-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32840/12804/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-199547/34234.0_TTbar_14TeV+2026D75+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-199547/34634.0_TTbar_14TeV+2026D76+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 37
• DQMHistoTests: Total histograms compared: 2751765
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2751742
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
• Checked 156 log files, 37 edm output root files, 37 DQM output files

[silviodonato]

@cms-sw/geometry-l2 @cms-sw/reconstruction-l2 do you have any comments?

[perrotta]

+1

• Only change for reco is the move to the new scenario in a test config
• Jenkins tests pass

[cvuosalo]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1

[makortel]

Could this PR be causing a failure in 33434.0 step 1? See #32882 for more details.

----- Begin Fatal Exception 11-Feb-2021 18:38:24 CET-----------------------
An exception of category 'DDException' occurred while
   [0] Processing global begin Run run: 1
   [1] Calling method for module OscarMTProducer/'g4SimHits'
   [2] Using EventSetup component XMLIdealGeometryESSource/'' to make data DDCompactView/'' in record IdealGeometryRecord
   [3] Running EventSetup component XMLIdealGeometryESSource/'
Exception Message:
ClhepEvaluator ERROR: can't evaluate: etl___ETLthickness!
 nmspace=
 varname=etl___ETLthickness
 exp=
  at=
----- End Fatal Exception -------------------------------------------------

[cvuosalo]

This PR makes any scenario that uses v5/etl.xml depend upon v5/caloBase.xml. However, scenarios D73 and earlier use v5/etl.xml and v3/caloBase.xml, which then creates the error because ZposAL used by v5/etl.xml from v5/caloBase.xml does not exist in v3/caloBase.xml.

Either this PR should be reverted, or this version mismatch should be fixed.

[srimanob]

Since we plan to keep old geometry, i.e. D72, should we make v6/etl.ml (instead of updating v5) to be used with v5/caloBase.xml? @fabiocos Thanks.