Added PPS Alignment to the Prompt Calibration Loop

[MatiXOfficial]

PR description:

The main purpose of this PR is to add some new files and modifications required to run the PPS alignment procedure in the PCL. The implementation is based on #33215. Note that the modifications include updating two files in CalibPPS/TimingCalibration.

It is important for us to be able to test it on Tier0 as soon as possible, so these changes are supposed to be included in CMSSW_12_1_0. Since the deadline for CMSSW_12_1_0_pre5 is 19/10, we kindly ask for a quick review.

This PR also improves PPSAlignmentWorker and PPSAlignmentHarvester. It applies a refactor, fixes some bugs and adds some new functionality to the code in the CalibPPS/AlignmentGlobal package. The main modifications are listed below:

• huge refactor in the worker and the harvester
• README update to include new parameters from the PPSAlignmentConfiguration conditions format
• test folder update
• Replaced PPSAlignmentConfig with PPSAlignmentConfiguration in the worker and the harvester.
• Added support for the new parameters from PPSAlignmentConfiguration.
• Now, the harvester can save an sqlite file with the alignment results.
• Updated the debug mode in the harvester to save more plots.
• Fixed odd fitting in the harvester (in the yAlignment method).

Apart from that, this PR adds a RetrieveCTPPSRPAlignmentCorrectionsData module required to properly test writing an sqlite file containing a CTPPSRPAlignmentCorrectionsData object.

The PR is a continuation of the strategy presented in the AlCaDB meeting on 30 August. The previous PR were:

• PPS: global alignment #31728 - first version of the code responsible for the PPS alignment,
• PPS Alignment - bug fix #34844 and PPS Alignment - bug fix (backport) #34845 - bug fix with backport,
• Added PPSAlignmentConfiguration - a new conditions format for PPS global alignment #35174 - new, final conditions object - PPSAlignmentConfiguration.

There are still some smaller changes foreseen, but they have been moved to the next PR, since it is vital to merge the PCL changes as soon as possible.

This PR also changes the online GTs, by including the new tags required in this PR and also get in line with the status of the 12_0_X currently used. Here are the diffs wrt the previous 12_1_X GTs:

• 121X_dataRun3_HLT_v6
• 121X_dataRun3_Express_v6
• 121X_dataRun3_Prompt_v6

And the diffs wrt to the 12_0_X GTs:

• 121X_dataRun3_HLT_v6
• 121X_dataRun3_Express_v6
• 121X_dataRun3_Prompt_v6

PR validation:

• The code returns correct results running on data from Run 2. Check out CalibPPS/AlignmentGlobal/test for an example.
• A new matrix test workflow has been added - 1042. It does not crash, but I've observed that the harvester (4th step) does not see any plots booked by the worker (3rd step). I don't know if that's because of a faulty configuration of the relval steps or wrong command used to run it (runTheMatrix.py -l 1042 --ibeos). Please, let me know if you know the answer to that. If not, I can remove that matrix test from this PR. By the way, I've observed the same problem while running the workflow 1041, which is very similar to 1042.
• To be able to run the matrix test, a candidate GT has been created - 121X_dataRun3_Prompt_Candidate_2021_10_09_12_20_39. For now, it is still listed as --conditions in relval_steps.py. Please, let me know if I have to replace it with auto:run3_data_express.
• In case they are needed, I attach the memory consumption plots acquired during running the procedure using a representative dataset from 2018.
  

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35631/25902

• This PR adds an extra 88KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/AlCa/python/autoPCL.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507
  • File Configuration/EventContent/python/AlCaRecoOutput_cff.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507
  • File Configuration/PyReleaseValidation/python/relval_production.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507
  • File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507, Use pdigi_hi_nogen for HI MC wfs #35028
  • File Configuration/StandardSequences/python/AlCaHarvesting_cff.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507
  • File Configuration/StandardSequences/python/AlCaRecoStreams_cff.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507

[cmsbuild]

A new Pull Request was created by @MatiXOfficial (Mateusz Kocot) for master.

It involves the following packages:

• CalibPPS/AlignmentGlobal (alca)
• CalibPPS/TimingCalibration (alca)
• CondTools/CTPPS (db)
• Configuration/AlCa (alca)
• Configuration/EventContent (operations)
• Configuration/PyReleaseValidation (pdmv, upgrade)
• Configuration/StandardSequences (operations)

@perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @wajidalikhan, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @ggovi, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@fabiocos, @makortel, @felicepantaleo, @fabferro, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @missirol, @ebrondol, @tocheng, @lecriste, @mmusich, @slomeo, @mtosi, @dgulhan, @kpedro88 this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[tvami]

Hi @MatiXOfficial please change the autoCond file to use the candidate GT so that we can trigger some Jenkins tests. After that's done, we'll make it a versioned GT. I have a question tho, PCL is run with the the Prompt GT so why you'd like to change auto:run3_data_express? It should be the prompt version, right?

[MatiXOfficial]

Thanks @tvami!
I'm a little confused now... I thought that PCL actually runs in express, and that's the reason for run3_data_express. @vavati, @jan-kaspar, can you take a look at this?
Anyway, I moved the candidate GT to run3_data_prompt in autoCond.py.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35631/25911

• This PR adds an extra 136KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/AlCa/python/autoCond.py modified in PR(s): Include realistic CSCBadChambers in Run-3 MC GTs #35637
  • File Configuration/AlCa/python/autoPCL.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507
  • File Configuration/EventContent/python/AlCaRecoOutput_cff.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507
  • File Configuration/PyReleaseValidation/python/relval_production.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507
  • File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507, Use pdigi_hi_nogen for HI MC wfs #35028
  • File Configuration/StandardSequences/python/AlCaHarvesting_cff.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507
  • File Configuration/StandardSequences/python/AlCaRecoStreams_cff.py modified in PR(s): Prompt calibration loop module for SiPixelLorentzAngle #35507

[cmsbuild]

Pull request #35631 was updated. @perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @wajidalikhan, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @ggovi, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please check and sign again.

[vavati]

@tvami Concerning the GT: the candidate was built vs the prompt version as the express_queue didn't include yet all the tags of PPS. There will be updates by @malbouis . For the time being to test this PR the prompt version should be ok.

[malbouis]

@MatiXOfficial, we (AlCaDB) suggest that you update autoCond.py with the following GT candidates and test your code with those. In case all goes well we will create versioned GTs and ask you to update autoCond.py with those.

They include all PPS tags needed for the proton reconstruction with the two PPS Alignment tags added in this PR, PPSAlignmentConfiguration_v1_express and PPSAlignmentConfiguration_reference_v1_express. Please note that the PPS tags included here will have to updated in the near future.

121X_dataRun3_Prompt_Candidate_2021_10_13_14_28_13
121X_dataRun3_HLT_Candidate_2021_10_13_14_29_46
121X_dataRun3_Express_Candidate_2021_10_13_14_32_06

For the relval steps, please use the Express GT.

[tvami]

+db

[srimanob]

+Upgrade

For the upgrade part, a new workflow 1042 is added with a new defined sequence. PR test runs fine.

[jordan-martins]

+1

[tvami]

@perrotta @qliphy this is essentially fully signed

[tvami]

urgent

[qliphy]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.