added --sites options for runTheMatrix #31535

smuzaffar · 2020-09-22T08:43:54Z

As requested here #22278 , added --sites <site> option to runTheMatrix.py script to select a specific site for recycle data. Setting it to emptry string will allow to search all sites.

This resolves #22278

cmsbuild · 2020-09-22T08:44:16Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-09-22T08:49:33Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31535/18518

This PR adds an extra 20KB to repository

cmsbuild · 2020-09-22T08:49:55Z

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@chayanit, @cmsbuild, @wajidalikhan, @kpedro88, @jordan-martins can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

smuzaffar · 2020-09-22T08:51:27Z

please test

cmsbuild · 2020-09-22T08:51:50Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-09-21-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/9483/console Started: 2020/09/22 10:53

cmsbuild · 2020-09-22T10:02:50Z

+1
Tested at: 684229d
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0d261f/9483/summary.html
CMSSW: CMSSW_11_2_X_2020-09-21-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-09-22T10:02:53Z

Comparison job queued.

cmsbuild · 2020-09-22T12:07:23Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0d261f/9483/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2540471
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2540442
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

kpedro88 · 2020-09-22T15:27:40Z

+upgrade

chayanit · 2020-09-22T15:44:36Z

Configuration/PyReleaseValidation/python/MatrixUtil.py

        if len(self.run) is not 0:
-            return ["file {0}={1} run={2} site=T2_CH_CERN".format(query_by, query_source, query_run) for query_run in self.run]
+            return ["file {0}={1} run={2}{3}".format(query_by, query_source, query_run, site) for query_run in self.run]
            #return ["file {0}={1} run={2} ".format(query_by, query_source, query_run) for query_run in self.run]


Do we still need to keep this line?

chayanit · 2020-09-22T15:45:05Z

Configuration/PyReleaseValidation/python/MatrixUtil.py

            #return ["file {0}={1} run={2} ".format(query_by, query_source, query_run) for query_run in self.run]
        else:
-            return ["file {0}={1} site=T2_CH_CERN".format(query_by, query_source)]
+            return ["file {0}={1}{2}".format(query_by, query_source, site)]
            #return ["file {0}={1} ".format(query_by, query_source)]


Do we still need to keep this line?

silviodonato · 2020-09-25T19:02:15Z

merge
we can clean MatrixUtil.py in a follow up PR

chayanit · 2020-10-08T15:04:58Z

Hello @smuzaffar, @silviodonato , this PR doesn't seem to remove a whitelist from CERN site. We have problem to read input for RelVals in 11_2_0_pre7. Can we revisit this?

smuzaffar · 2020-10-08T15:15:57Z

@chayanit , I did not want to change the default behavior of the script. So by default CERN site is selected. As mentioned int he description of the PR, running --site='' should allow to search all sites. If this behavior not working?

chayanit · 2020-10-08T15:21:06Z

@smuzaffar seems not, apparently it always looks for sample at CERN site and not others

smuzaffar · 2020-10-08T16:52:51Z

It works for me e.g running without --site option I see that runTheMatrix runs the following

>runTheMatrix.py -i all -l 1000 --maxSteps=1 --dryRun
dasgoclient --limit 0 --query 'file dataset=/MinimumBias/Run2011A-v1/RAW run=165121 site=T2_CH_CERN'

and with --site='' i get

>runTheMatrix.py -i all -l 1000 --maxSteps=1 --dryRun --site=''
 dasgoclient --limit 0 --query 'file dataset=/MinimumBias/Run2011A-v1/RAW run=165121'

chayanit · 2020-10-09T08:01:20Z

Hi @smuzaffar , it doesn't work for us and we found this is caused by the default setting here

cmssw/Configuration/PyReleaseValidation/scripts/runTheMatrix.py

Line 287 in a65b92a

default='T2_CH_CERN',

smuzaffar · 2020-10-09T08:05:14Z

How can I reproduce it?

smuzaffar · 2020-10-09T08:07:31Z

are you calling runTheMatrix to generate the configuration or importing the MatrixUtils directly? If you are not using runTheMatrix then I would suggest to set environment variable CMSSW_DAS_QUERY_SITES=''

chayanit · 2020-10-09T08:08:12Z

Yes we run runTheMatrix to generate configuration

smuzaffar · 2020-10-09T08:11:32Z

Can you please share the runTheMatrix.py command you run?

chayanit · 2020-10-09T08:28:36Z

Sure, this is the command line we usually run for RelVal production
runTheMatrix.py --what upgrade -l 11834.0 -t 4 -m 11500 -b 'fullsim_PU_2021_14TeV' -i all --noCaf --wm force
while you need to update string here https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/python/relval_steps.py#L3211 to "CMSSW_11_2_0_pre7-112X_mcRun3_2021_realistic_v8-v"

smuzaffar · 2020-10-09T08:35:05Z

I do not see --sites='' option here, so for sure it will use the default. As I have mentioned earlier that by default runTheMatrix.py uses T2_CH_CERN , this was the behavior for many year and I did not want to change that. That is why I added the extra command line option. In case you want to remove the T2_CH_CERN from the whitelist then please call runTheMatrix.py with --sites='' command-line option

chayanit · 2020-10-09T08:46:33Z

Ah ok @smuzaffar you mean we have to put --sites='' explicitly in the command line?

smuzaffar · 2020-10-09T08:48:47Z

yes

chayanit · 2020-10-09T08:49:27Z

Can you show how explicitly?

smuzaffar · 2020-10-09T08:49:49Z

runTheMatrix.py --what upgrade -l 11834.0 -t 4 -m 11500 -b 'fullsim_PU_2021_14TeV' -i all --noCaf --wm force --sites=''

see the extra --sites='' at the end

chayanit · 2020-10-09T08:51:29Z

this option is not shown in runTheMatrix.py -h though should we include?

smuzaffar · 2020-10-09T08:53:53Z

are you sure that you are using a release where this option is available? If I run runTheMatrix.py --help in CMSSW_11_2_X then I see

  --sites=DASSITES      Run DAS query to get data from a specific site
                        (default is T2_CH_CERN). Set it to empty string to
                        search all sites.

chayanit · 2020-10-09T09:00:52Z

Yeah now I saw the option is added to runTheMatrix.py script but it doesn't show up when run 'runTheMatrix.py --help' I'm doing it on 11_2_0_pre7

davidlange6 · 2020-10-09T09:06:54Z

Seems to work for me [dlange@lxplus757 dlange]$ scram p CMSSW_11_2_0_pre7 [dlange@lxplus757 dlange]$ cd CMSSW_11_2_0_pre7 [dlange@lxplus757 CMSSW_11_2_0_pre7]$ cmsenv [dlange@lxplus757 CMSSW_11_2_0_pre7]$ runTheMatrix.py -h | grep -A3 site --ibeos Use IB EOS site configuration --sites=DASSITES Run DAS query to get data from a specific site (default is T2_CH_CERN). Set it to empty string to search all sites.

…

On Oct 9, 2020, at 11:01 AM, Chayanit Asawatangtrakuldee ***@***.***> wrote: Yeah now I saw the option is added to runTheMatrix.py script but it doesn't show up when run 'runTheMatrix.py --help' I'm doing it on 11_2_0_pre7 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

added --sites options for runTheMatrix

684229d

cmsbuild added this to the CMSSW_11_2_X milestone Sep 22, 2020

cmsbuild added code-checks-pending comparison-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Sep 22, 2020

smuzaffar mentioned this pull request Sep 22, 2020

runTheMatrix.py -i all whitelists CERN even if files are elsewhere #22278

Closed

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 22, 2020

cmsbuild added tests-started and removed tests-pending labels Sep 22, 2020

cmsbuild added tests-approved and removed tests-started labels Sep 22, 2020

cmsbuild added comparison-available and removed comparison-pending labels Sep 22, 2020

cmsbuild added upgrade-approved and removed upgrade-pending labels Sep 22, 2020

chayanit reviewed Sep 22, 2020

View reviewed changes

cmsbuild added orp-approved and removed orp-pending labels Sep 25, 2020

cmsbuild merged commit a65b92a into cms-sw:master Sep 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added --sites options for runTheMatrix #31535

added --sites options for runTheMatrix #31535

smuzaffar commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

smuzaffar commented Sep 22, 2020

cmsbuild commented Sep 22, 2020 •

edited

cmsbuild commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

kpedro88 commented Sep 22, 2020

chayanit Sep 22, 2020

chayanit Sep 22, 2020

silviodonato commented Sep 25, 2020

chayanit commented Oct 8, 2020

smuzaffar commented Oct 8, 2020

chayanit commented Oct 8, 2020

smuzaffar commented Oct 8, 2020 •

edited

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020

smuzaffar commented Oct 9, 2020 •

edited

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020 •

edited

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020 •

edited

chayanit commented Oct 9, 2020

davidlange6 commented Oct 9, 2020 via email

added --sites options for runTheMatrix #31535

added --sites options for runTheMatrix #31535

Conversation

smuzaffar commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

smuzaffar commented Sep 22, 2020

cmsbuild commented Sep 22, 2020 • edited

cmsbuild commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

cmsbuild commented Sep 22, 2020

kpedro88 commented Sep 22, 2020

chayanit Sep 22, 2020

Choose a reason for hiding this comment

chayanit Sep 22, 2020

Choose a reason for hiding this comment

silviodonato commented Sep 25, 2020

chayanit commented Oct 8, 2020

smuzaffar commented Oct 8, 2020

chayanit commented Oct 8, 2020

smuzaffar commented Oct 8, 2020 • edited

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020

smuzaffar commented Oct 9, 2020 • edited

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020 • edited

chayanit commented Oct 9, 2020

smuzaffar commented Oct 9, 2020 • edited

chayanit commented Oct 9, 2020

davidlange6 commented Oct 9, 2020 via email

cmsbuild commented Sep 22, 2020 •

edited

smuzaffar commented Oct 8, 2020 •

edited

smuzaffar commented Oct 9, 2020 •

edited

smuzaffar commented Oct 9, 2020 •

edited

smuzaffar commented Oct 9, 2020 •

edited