Add new uploader script to CMSSW #33479

Dres90 · 2021-04-20T15:04:11Z

PR description:

Include new uploader script for the V2 version of the CondDbUploader to be distributed via CMSSW

cmsbuild · 2021-04-20T15:05:08Z

A new Pull Request was created by @Dres90 (Andrés Cárdenas) for CMSSW_11_3_X.

It involves the following packages:

CondCore/Utilities

@ggovi, @cmsbuild can you please review it and eventually sign? Thanks.
@mmusich this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2021-04-20T15:17:12Z

Pull request #33479 was updated. @ggovi, @cmsbuild can you please check and sign again.

ggovi · 2021-04-20T15:25:40Z

please test

tvami · 2021-04-20T15:57:31Z

Hi @Dres90 , thanks a lot for this new uploader script. If you remember at the AlCaDB meeting when you presented about this, I asked if it is possible to do Prep-To-Prod direct uploads (without having to download a local sqlite file from Prep and upload that to Prod) with the new version of the code. I'm wondering how one does that with this new script. Thanks!

Dres90 · 2021-04-20T16:28:10Z

Hi @Dres90 , thanks a lot for this new uploader script. If you remember at the AlCaDB meeting when you presented about this, I asked if it is possible to do Prep-To-Prod direct uploads (without having to download a local sqlite file from Prep and upload that to Prod) with the new version of the code. I'm wondering how one does that with this new script. Thanks!

Hi @tvami, I do have that noted as a future improvement but currently it is not a functionality offered by the new uploader. Definitely a possibility but we want to get the stable version fully tested before beginning work in improvements.

tvami · 2021-04-20T17:04:10Z

Hi @Dres90 , thanks a lot for this new uploader script. If you remember at the AlCaDB meeting when you presented about this, I asked if it is possible to do Prep-To-Prod direct uploads (without having to download a local sqlite file from Prep and upload that to Prod) with the new version of the code. I'm wondering how one does that with this new script. Thanks!

Hi @tvami, I do have that noted as a future improvement but currently it is not a functionality offered by the new uploader. Definitely a possibility but we want to get the stable version fully tested before beginning work in improvements.

Ok, looking forward to it!

cmsbuild · 2021-04-20T18:11:01Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-05bb4a/14356/summary.html
COMMIT: f9f3ef9
CMSSW: CMSSW_11_3_X_2021-04-20-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33479/14356/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

Run3-gex70 Backport 2 PR's #33423 and #33424 to update 2021, 2026 scenarios #33455 @bsunanda: Run3-gex70 Backport 2 PR's Phase2-gex67 Add a new workflow with V15 version of HGCal Geometry #33423 and Run3-gex67 Modify Run3 workflows with changed version of RP devices #33424 to update 2021, 2026 scenarios
TkDQM: backport of PR 33430 #33463 @sroychow: TkDQM: backport of PR 33430

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-05bb4a/14356/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-05bb4a/14356/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2877046
DQMHistoTests: Total failures: 16
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2877007
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

tvami · 2021-04-20T18:27:15Z

CondCore/Utilities/scripts/uploadConditions_v2.py

+                '\nWhich is the destination database where the tags should be exported? \nPossible choices: oracle://cms_orcon_prod/CMS_CONDITIONS (for prod) or oracle://cms_orcoff_prep/CMS_CONDITIONS (for prep) \ndestinationDatabase: '
+            elif ntry==1:
+                inputMessage = \
+                '\nPlease choose one of the two valid destinations: \noracle://cms_orcon_prod/CMS_CONDITIONS (for prod) or oracle://cms_orcoff_prep/CMS_CONDITIONS (for prep) \


Would it be possible to simplify this and ask the user to choose between "Prep" or "Prod" instead of the full string of "oracle://cms_orcon_prod/CMS_CONDITIONS" or "oracle://cms_orcoff_prep/CMS_CONDITIONS"? Or even 0 and 1, the same way when we choose the tags? That would make the user's life a bit more convenient. Thanks!

Thank you for the suggestion, it is a great idea.
I have just added that functionality and updated the PR. The suggestions to the user are oradev and orapro to keep consistency with the conddb tools, however it will also resolve to the proper destination if you input "prep" or "prod".

Awesome, thanks!

…lection based on destination database

cmsbuild · 2021-04-21T09:24:24Z

Pull request #33479 was updated. @ggovi, @cmsbuild can you please check and sign again.

Dres90 · 2021-04-21T09:28:13Z

PR has been updated with the following changes:

Wizard allows user to choose destination database by inputting oradev, oraprod, dev or prod, simplifying the user experience.
When no server is specified, the script chooses the appropriate server based on the destination database.

ggovi · 2021-04-21T13:31:58Z

please test

cmsbuild · 2021-04-21T16:25:17Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-05bb4a/14385/summary.html
COMMIT: 4b799df
CMSSW: CMSSW_11_3_X_2021-04-20-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33479/14385/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2877046
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2877017
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

ggovi · 2021-04-22T07:04:18Z

+1

cmsbuild · 2021-04-22T07:04:42Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_11_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

mmusich · 2021-04-22T07:06:45Z

@ggovi @Dres90 out of curiosity, are users invited to use this in lieu of the old script, once it gets distributed to cmssw?

ggovi · 2021-04-22T15:03:27Z

@mmusich good point. At some point, that 'invitation' will be explicit. We are still somewhat in the introductory phase...

qliphy · 2021-04-23T08:56:12Z

@Dres90 Please make a PR first to master before backporting to 11_3_X.

Dres90 · 2021-04-23T09:28:26Z

Closing this PR as it was pointing to a prior release. Replaced by
#33511 now pointing to master branch.

Dres90 added 3 commits April 20, 2021 16:55

Add new uploader script

1ccb324

Revert commit

802f7c6

Add new uploader script

c2c5864

cmsbuild added this to the CMSSW_11_3_X milestone Apr 20, 2021

cmsbuild added db-pending orp-pending pending-signatures tests-pending labels Apr 20, 2021

Update script header

f9f3ef9

cmsbuild added tests-started and removed tests-pending labels Apr 20, 2021

cmsbuild added tests-approved and removed tests-started labels Apr 20, 2021

tvami reviewed Apr 20, 2021

View reviewed changes

Improve selection for destination databse and add automatic server se…

4b799df

…lection based on destination database

cmsbuild added tests-pending and removed tests-approved labels Apr 21, 2021

cmsbuild added tests-started and removed tests-pending labels Apr 21, 2021

cmsbuild added tests-approved and removed tests-started labels Apr 21, 2021

cmsbuild added db-approved fully-signed and removed db-pending pending-signatures labels Apr 22, 2021

Dres90 mentioned this pull request Apr 23, 2021

Add new uploader script to CMSSW to master #33511

Merged

Dres90 closed this Apr 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new uploader script to CMSSW #33479

Add new uploader script to CMSSW #33479

Dres90 commented Apr 20, 2021

cmsbuild commented Apr 20, 2021

cmsbuild commented Apr 20, 2021

ggovi commented Apr 20, 2021

tvami commented Apr 20, 2021

Dres90 commented Apr 20, 2021

tvami commented Apr 20, 2021

cmsbuild commented Apr 20, 2021

tvami Apr 20, 2021 •

edited

Dres90 Apr 21, 2021

tvami Apr 21, 2021

cmsbuild commented Apr 21, 2021

Dres90 commented Apr 21, 2021

ggovi commented Apr 21, 2021

cmsbuild commented Apr 21, 2021

ggovi commented Apr 22, 2021

cmsbuild commented Apr 22, 2021

mmusich commented Apr 22, 2021

ggovi commented Apr 22, 2021

qliphy commented Apr 23, 2021

Dres90 commented Apr 23, 2021

Add new uploader script to CMSSW #33479

Add new uploader script to CMSSW #33479

Conversation

Dres90 commented Apr 20, 2021

PR description:

cmsbuild commented Apr 20, 2021

cmsbuild commented Apr 20, 2021

ggovi commented Apr 20, 2021

tvami commented Apr 20, 2021

Dres90 commented Apr 20, 2021

tvami commented Apr 20, 2021

cmsbuild commented Apr 20, 2021

Comparison Summary

tvami Apr 20, 2021 • edited

Choose a reason for hiding this comment

Dres90 Apr 21, 2021

Choose a reason for hiding this comment

tvami Apr 21, 2021

Choose a reason for hiding this comment

cmsbuild commented Apr 21, 2021

Dres90 commented Apr 21, 2021

ggovi commented Apr 21, 2021

cmsbuild commented Apr 21, 2021

Comparison Summary

ggovi commented Apr 22, 2021

cmsbuild commented Apr 22, 2021

mmusich commented Apr 22, 2021

ggovi commented Apr 22, 2021

qliphy commented Apr 23, 2021

Dres90 commented Apr 23, 2021

tvami Apr 20, 2021 •

edited