[12.3.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38188

mmusich · 2022-06-01T21:19:16Z

backport of #38186

PR description:

It was pointed out in https://cms-talk.web.cern.ch/t/alcaprompt-datasets-not-loaded-in-dbs/11146/1 that, quoting:

Since the last update of the alcareco matrix, two new producers were added to ExpressPhysics:

PromptCalibProdSiPixelLorentzAngle

PromptCalibProdSiStripHitEfficiency
while the resulting (harvested) ROOT files are regularly displayed in the DQM, the corresponding ALCAPROMPT datasets in DAS are missing.

The names of the datasets expected are:

/StreamExpress/Run2022A-PromptCalibProdSiStripHitEfficiency-Express-v1/ALCAPROMPT
/StreamExpress/Run2022A-PromptCalibProdSiPixelLorentzAngle-Express-v1/ALCAPROMPT

According to Tier0 experts the problem is related to the name of the dataset, which is too long to be accepted, and the WMAgent software at present is not signalling this but keeps trying the upload. While probably a better handling of this issue should be thought for future, we need to fix the cause of the problem by shortening the names.

This PR performs such shortening:

ALCARECOPromptCalibProdSiPixelLorentzAngle is renamed ALCARECOPromptCalibProdSiPixelLA
ALCARECOStreamPromptCalibProdSiStripHitEfficiency is renamed ALCARECOStreamPromptCalibProdSiStripHitEff

PR validation:

Run the following commands:

cmsDriver.py testReAlCa -s ALCA:PromptCalibProdSiPixelLA --conditions 123X_dataRun3_Express_v6 --scenario pp --data --era Run2_2018 --datatier ALCARECO --eventcontent ALCARECO --processName=ReAlCa -n 10000 --dasquery='file dataset=/StreamExpress/Run2022A-SiPixelCalSingleMuon-Express-v1/ALCARECO run=352572' --customise_commands='process.ALCARECOCalSignleMuonFilterForSiPixelLorentzAngle.TriggerResultsTag = cms.InputTag ( "TriggerResults","","HLT" ) ; process.ALCARECOCalSignleMuonFilterForSiPixelLorentzAngle.HLTPaths = ["*"]' --nThreads=4

cmsDriver.py stepHarvest -s ALCAHARVEST:SiPixelLA --conditions 123X_dataRun3_Express_v6  --scenario pp --data --era Run2_2018 --filein file:PromptCalibProdSiPixelLA.root -n -1

and

cmsDriver.py testReAlCa -s ALCA:PromptCalibProdSiStripHitEff --conditions 123X_dataRun2_v2 --scenario pp --data --era Run2_2018 --datatier ALCARECO --eventcontent ALCARECO --processName=ReAlCa -n 100000 --dasquery='file dataset=/StreamExpress/Run2018D-SiStripCalMinBias-Express-v1/ALCARECO run=325172' --nThreads=4

msDriver.py stepHarvest -s ALCAHARVEST:SiStripHitEff --conditions 123X_dataRun2_v2 --scenario pp --data --era Run2_2018 --filein file:PromptCalibProdSiStripHitEff.root -n -1

Also exercised successfully the two runTheMatrix workflows running the PCL workflows:

runTheMatrix.py -l 1001,1001.2

if this PR is a backport please specify the original PR and why you need to backport that PR:

Verbatim backport of #38186

…treamPromptCalibProdSiStripHitEff and ALCARECOStreamPromptCalibProdSiPixelLorentzAngle -> ALCARECOStreamPromptCalibProdSiPixelLA

cmsbuild · 2022-06-01T21:19:38Z

A new Pull Request was created by @mmusich (Marco Musich) for CMSSW_12_3_X.

It involves the following packages:

Calibration/TkAlCaRecoProducers (alca)
Configuration/PyReleaseValidation (pdmv, upgrade)
Configuration/StandardSequences (operations)

@perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@mtosi, @beaucero, @fabiocos, @makortel, @felicepantaleo, @slomeo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @missirol, @ebrondol, @tocheng, @lecriste, @trtomei, @mmusich, @threus, @dgulhan, @kpedro88 this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Backported from Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38186

mmusich · 2022-06-02T05:54:58Z

test parameters:

workflow = 1001.2

mmusich · 2022-06-02T05:55:12Z

please test

cmsbuild · 2022-06-02T09:50:17Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ad0972/25190/summary.html
COMMIT: 69839c2
CMSSW: CMSSW_12_3_X_2022-06-01-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38188/25190/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ad0972/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3627570
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3627540
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 210 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

cmsbuild · 2022-06-02T13:56:33Z

Pull request #38188 was updated. @perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please check and sign again.

mmusich · 2022-06-02T13:57:02Z

please test

mmusich · 2022-06-02T14:36:50Z

the unit test failure is unrelated.

cmsbuild · 2022-06-02T17:52:42Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ad0972/25202/summary.html
COMMIT: f86a438
CMSSW: CMSSW_12_3_X_2022-06-02-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38188/25202/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test-das-selected-lumis had ERRORS

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ad0972/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3627570
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3627546
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 210 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

mmusich · 2022-06-02T21:59:48Z

@cmsbuild, please test

let's try again

cmsbuild · 2022-06-03T01:36:27Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ad0972/25229/summary.html
COMMIT: f86a438
CMSSW: CMSSW_12_3_X_2022-06-02-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38188/25229/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ad0972/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3627570
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3627540
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 210 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

fabiocos · 2022-06-03T08:03:28Z

urgent

the absence of this fix is creating a backlog of not properly uploaded files in Tier0, which should be stopped asap

srimanob · 2022-06-03T08:04:45Z

+Upgrade

Backport PR.

mmusich · 2022-06-03T10:47:06Z

sorry to constantly ping you @cms-sw/pdmv-l2 :)

kskovpen · 2022-06-03T10:59:57Z

Sorry for delay

kskovpen · 2022-06-03T11:00:03Z

+pdmv

mmusich · 2022-06-03T11:01:52Z

thanks @kskovpen see also the 12.4.X version of this #38187

francescobrivio · 2022-06-03T11:33:59Z

@cms-sw/orp-l2 this is basically fully signed now, just orp and operations missing!

qliphy · 2022-06-03T11:39:47Z

+1

cmsbuild · 2022-06-03T11:40:08Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will be automatically merged.

rename ALCARECOStreamPromptCalibProdSiStripHitEfficiency -> ALCARECOS…

69839c2

…treamPromptCalibProdSiStripHitEff and ALCARECOStreamPromptCalibProdSiPixelLorentzAngle -> ALCARECOStreamPromptCalibProdSiPixelLA

cmsbuild added this to the CMSSW_12_3_X milestone Jun 1, 2022

cmsbuild added alca-pending backport operations-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Jun 1, 2022

cmsbuild added tests-started and removed tests-pending labels Jun 2, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 2, 2022

mmusich added 2 commits June 2, 2022 15:55

update autoPCL to reflect the new naming scheme

aade899

rename also the Output commands

f86a438

cmsbuild added tests-pending and removed tests-approved labels Jun 2, 2022

cmsbuild added tests-started and removed tests-pending labels Jun 2, 2022

cmsbuild added tests-rejected and removed tests-started labels Jun 2, 2022

cmsbuild added backport-ok and removed backport labels Jun 2, 2022

cmsbuild added tests-started and removed tests-rejected labels Jun 2, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 3, 2022

cmsbuild added the urgent label Jun 3, 2022

cmsbuild added upgrade-approved and removed upgrade-pending labels Jun 3, 2022

cmsbuild added pdmv-approved and removed pdmv-pending labels Jun 3, 2022

cmsbuild added fully-signed operations-approved orp-approved and removed operations-pending pending-signatures orp-pending labels Jun 3, 2022

cmsbuild merged commit 7a5efc7 into cms-sw:CMSSW_12_3_X Jun 3, 2022

mmusich deleted the renameLongAlCaStreamName_12_3_X branch June 3, 2022 11:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[12.3.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38188

[12.3.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38188

mmusich commented Jun 1, 2022

cmsbuild commented Jun 1, 2022 •

edited

mmusich commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

mmusich commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 3, 2022

fabiocos commented Jun 3, 2022

srimanob commented Jun 3, 2022

mmusich commented Jun 3, 2022

kskovpen commented Jun 3, 2022

kskovpen commented Jun 3, 2022

mmusich commented Jun 3, 2022

francescobrivio commented Jun 3, 2022

qliphy commented Jun 3, 2022

cmsbuild commented Jun 3, 2022

[12.3.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38188

[12.3.X] Shorten the names of two ALCARECOStreams in order to fit into DBS database schema limitations #38188

Conversation

mmusich commented Jun 1, 2022

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Jun 1, 2022 • edited

mmusich commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

Comparison Summary

cmsbuild commented Jun 2, 2022

mmusich commented Jun 2, 2022

mmusich commented Jun 2, 2022

cmsbuild commented Jun 2, 2022

Unit Tests

Comparison Summary

mmusich commented Jun 2, 2022

cmsbuild commented Jun 3, 2022

Comparison Summary

fabiocos commented Jun 3, 2022

srimanob commented Jun 3, 2022

mmusich commented Jun 3, 2022

kskovpen commented Jun 3, 2022

kskovpen commented Jun 3, 2022

mmusich commented Jun 3, 2022

francescobrivio commented Jun 3, 2022

qliphy commented Jun 3, 2022

cmsbuild commented Jun 3, 2022

cmsbuild commented Jun 1, 2022 •

edited