[12.4.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38187

mmusich · 2022-06-01T21:19:14Z

backport of #38186

PR description:

It was pointed out in https://cms-talk.web.cern.ch/t/alcaprompt-datasets-not-loaded-in-dbs/11146/1 that, quoting:

Since the last update of the alcareco matrix, two new producers were added to ExpressPhysics:

PromptCalibProdSiPixelLorentzAngle

PromptCalibProdSiStripHitEfficiency
while the resulting (harvested) ROOT files are regularly displayed in the DQM, the corresponding ALCAPROMPT datasets in DAS are missing.

The names of the datasets expected are:

/StreamExpress/Run2022A-PromptCalibProdSiStripHitEfficiency-Express-v1/ALCAPROMPT
/StreamExpress/Run2022A-PromptCalibProdSiPixelLorentzAngle-Express-v1/ALCAPROMPT

According to Tier0 experts the problem is related to the name of the dataset, which is too long to be accepted, and the WMAgent software at present is not signalling this but keeps trying the upload. While probably a better handling of this issue should be thought for future, we need to fix the cause of the problem by shortening the names.

This PR performs such shortening:

ALCARECOPromptCalibProdSiPixelLorentzAngle is renamed ALCARECOPromptCalibProdSiPixelLA
ALCARECOStreamPromptCalibProdSiStripHitEfficiency is renamed ALCARECOStreamPromptCalibProdSiStripHitEff

PR validation:

Run the following commands:

cmsDriver.py testReAlCa -s ALCA:PromptCalibProdSiPixelLA --conditions 123X_dataRun3_Express_v6 --scenario pp --data --era Run2_2018 --datatier ALCARECO --eventcontent ALCARECO --processName=ReAlCa -n 10000 --dasquery='file dataset=/StreamExpress/Run2022A-SiPixelCalSingleMuon-Express-v1/ALCARECO run=352572' --customise_commands='process.ALCARECOCalSignleMuonFilterForSiPixelLorentzAngle.TriggerResultsTag = cms.InputTag ( "TriggerResults","","HLT" ) ; process.ALCARECOCalSignleMuonFilterForSiPixelLorentzAngle.HLTPaths = ["*"]' --nThreads=4

cmsDriver.py stepHarvest -s ALCAHARVEST:SiPixelLA --conditions 123X_dataRun3_Express_v6  --scenario pp --data --era Run2_2018 --filein file:PromptCalibProdSiPixelLA.root -n -1

and

cmsDriver.py testReAlCa -s ALCA:PromptCalibProdSiStripHitEff --conditions 123X_dataRun2_v2 --scenario pp --data --era Run2_2018 --datatier ALCARECO --eventcontent ALCARECO --processName=ReAlCa -n 100000 --dasquery='file dataset=/StreamExpress/Run2018D-SiStripCalMinBias-Express-v1/ALCARECO run=325172' --nThreads=4

msDriver.py stepHarvest -s ALCAHARVEST:SiStripHitEff --conditions 123X_dataRun2_v2 --scenario pp --data --era Run2_2018 --filein file:PromptCalibProdSiStripHitEff.root -n -1

Also exercised successfully the two runTheMatrix workflows running the PCL workflows:

runTheMatrix.py -l 1001,1001.2

if this PR is a backport please specify the original PR and why you need to backport that PR:

Verbatim backport of #38186

…treamPromptCalibProdSiStripHitEff and ALCARECOStreamPromptCalibProdSiPixelLorentzAngle -> ALCARECOStreamPromptCalibProdSiPixelLA

cmsbuild · 2022-06-01T21:19:33Z

A new Pull Request was created by @mmusich (Marco Musich) for CMSSW_12_4_X.

It involves the following packages:

Calibration/TkAlCaRecoProducers (alca)
Configuration/PyReleaseValidation (pdmv, upgrade)
Configuration/StandardSequences (operations)

@perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@mtosi, @beaucero, @fabiocos, @makortel, @felicepantaleo, @slomeo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @missirol, @ebrondol, @tocheng, @lecriste, @trtomei, @mmusich, @threus, @dgulhan, @kpedro88 this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Backported from Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38186

mmusich · 2022-06-02T05:53:56Z

test parameters:

workflow = 1001.2

mmusich · 2022-06-02T05:54:03Z

please test

cmsbuild · 2022-06-02T09:56:47Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3fbeb8/25189/summary.html
COMMIT: 0b894a9
CMSSW: CMSSW_12_4_X_2022-06-01-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38187/25189/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-3fbeb8/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3670175
DQMHistoTests: Total failures: 13
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3670139
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 50 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 213 log files, 45 edm output root files, 51 DQM output files
TriggerResults: no differences found

cmsbuild · 2022-06-02T13:55:34Z

Pull request #38187 was updated. @perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please check and sign again.

mmusich · 2022-06-02T13:56:17Z

@cmsbuild, please test

mmusich · 2022-06-02T14:35:55Z

the unit test failure is unrelated.

cmsbuild · 2022-06-02T18:19:21Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3fbeb8/25201/summary.html
COMMIT: fd82c9f
CMSSW: CMSSW_12_4_X_2022-06-02-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38187/25201/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test-das-selected-lumis had ERRORS

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-3fbeb8/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3670175
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3670151
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 213 log files, 45 edm output root files, 51 DQM output files
TriggerResults: no differences found

tvami · 2022-06-02T18:22:50Z

+alca

unit test failure is not connected

srimanob · 2022-06-02T19:12:47Z

+Upgrade

Backport PR.

srimanob · 2022-06-02T19:13:04Z

@cmsbuild please test

Just to clean up the failure.

cmsbuild · 2022-06-02T23:32:53Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3fbeb8/25222/summary.html
COMMIT: fd82c9f
CMSSW: CMSSW_12_4_X_2022-06-02-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38187/25222/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-3fbeb8/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3670175
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3670145
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
Checked 213 log files, 45 edm output root files, 51 DQM output files
TriggerResults: no differences found

kskovpen · 2022-06-03T11:57:02Z

+pdmv

francescobrivio · 2022-06-03T11:59:09Z

@cms-sw/orp-l2 this is basically fully signed as well!

qliphy · 2022-06-05T12:38:29Z

+1

cmsbuild · 2022-06-05T12:38:55Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will be automatically merged.

rename ALCARECOStreamPromptCalibProdSiStripHitEfficiency -> ALCARECOS…

0b894a9

…treamPromptCalibProdSiStripHitEff and ALCARECOStreamPromptCalibProdSiPixelLorentzAngle -> ALCARECOStreamPromptCalibProdSiPixelLA

cmsbuild added this to the CMSSW_12_4_X milestone Jun 1, 2022

cmsbuild added alca-pending backport operations-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Jun 1, 2022

cmsbuild added tests-started and removed tests-pending labels Jun 2, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 2, 2022

mmusich added 2 commits June 2, 2022 15:54

update autoPCL to reflect the new naming scheme

1d3eec5

rename also the Output commands

fd82c9f

cmsbuild added tests-pending and removed tests-approved labels Jun 2, 2022

cmsbuild added tests-started and removed tests-pending labels Jun 2, 2022

cmsbuild added tests-rejected and removed tests-started labels Jun 2, 2022

cmsbuild added alca-approved and removed alca-pending labels Jun 2, 2022

cmsbuild added upgrade-approved tests-started backport-ok and removed upgrade-pending tests-rejected backport labels Jun 2, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 2, 2022

mmusich mentioned this pull request Jun 3, 2022

[12.3.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38188

Merged

cmsbuild added pdmv-approved and removed pdmv-pending labels Jun 3, 2022

cmsbuild added fully-signed operations-approved orp-approved and removed operations-pending pending-signatures orp-pending labels Jun 5, 2022

cmsbuild merged commit 086b9ae into cms-sw:CMSSW_12_4_X Jun 5, 2022

mmusich deleted the renameLongAlCaStreamName_12_4_X branch June 5, 2022 12:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[12.4.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38187

[12.4.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38187

mmusich commented Jun 1, 2022

cmsbuild commented Jun 1, 2022 •

edited

mmusich commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

mmusich commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

tvami commented Jun 2, 2022

srimanob commented Jun 2, 2022

srimanob commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

kskovpen commented Jun 3, 2022

francescobrivio commented Jun 3, 2022

qliphy commented Jun 5, 2022

cmsbuild commented Jun 5, 2022

[12.4.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38187

[12.4.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38187

Conversation

mmusich commented Jun 1, 2022

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Jun 1, 2022 • edited

mmusich commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

Comparison Summary

cmsbuild commented Jun 2, 2022

mmusich commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

Unit Tests

Comparison Summary

tvami commented Jun 2, 2022

srimanob commented Jun 2, 2022

srimanob commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

Comparison Summary

kskovpen commented Jun 3, 2022

francescobrivio commented Jun 3, 2022

qliphy commented Jun 5, 2022

cmsbuild commented Jun 5, 2022

cmsbuild commented Jun 1, 2022 •

edited