runTheMatrix.py -i all whitelists CERN even if files are elsewhere

[wmtford]

Some workflows in runTheMatrix.py with the recycle option seek a GEN_SIM dataset that exists only at FNAL. The dasquery step fails because the command contains the requirement site=T2_CH_CERN. Could a command-line option --allsites, valid in conjunction with -i, be added to runTheMatrix.py that would allow the user of suppress this site requirement?

For example, in CMSSW_10_0_0_pre2 the command

runTheMatrix.py --command=--number=10 -w 2017 -l 10859.0 -i all

generates at step1
dasgoclient --limit 0 --query 'file dataset=/RelValQCD_Pt_3000_3500_13/CMSSW_10_0_0_pre2-100X_upgrade2018_realistic_v1-v1/GEN-SIM site=T2_CH_CERN'

But looking in DAS I find that this dataset exists only at T1_US_FNAL.

The query is generated in

Configuration/PyReleaseValidation/python/MatrixUtil.py

where we find

    if len(self.run) is not 0:
        return ["file {0}={1} run={2} site=T2_CH_CERN".format(query_by, query_source, query_run) for query_run in self.run]
        # return ["file {0}={1} run={2} ".format(query_by, query_source, query_run) for query_run in self.run]
    else:
        return ["file {0}={1} site=T2_CH_CERN".format(query_by, query_source)]
        # return ["file {0}={1} ".format(query_by, query_source)]

I can run successfully by reversing the comments in these two pairs of lines.

[cmsbuild]

A new Issue was created by @wmtford Bill Ford.

@davidlange6, @Dr15Jones, @smuzaffar, @fabiocos can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[fabiocos]

assign pdmv

[cmsbuild]

New categories assigned: pdmv

@fabozzi,@prebello,@GurpreetSinghChahal you have been requested to review this Pull request/Issue and eventually sign? Thanks

[prebello]

@fabiocos PdmV has been dealing with it so far. It is an eventual issue, not always happening.
Therefore PdmV doesn't think that this PR should be considered.

[prebello]

-1

[prebello]

@davidlange6, @Dr15Jones, @smuzaffar, @fabiocos
Could you please refresh my mind about the reason to maintain T2_CH_CERN in the whitelist as it is?
It seems that years ago @hengne has requested to change it in the past, receiving complains from offline for IB and PR tests.

[fabozzi]

Hi, the command line option --ibeos introduced recently by @smuzaffar with PR #22072
actually should do the work. Right?

[christopheralanwest]

I would like to note that an even more common command does not work as intended:

runTheMatrix.py -l limited -i all

which is suggested as a test before submitting any PR. As a result, a user cannot reproduce all of the PR integration tests. As an example from a recent PR, the workflow

4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC

executes

dasgoclient --limit 0 --query 'file dataset=/Cosmics/Run2011A-v1/RAW run=160960 site=T2_CH_CERN

which gives no results when run by a user, but somehow the same command returns a non-zero list of files when run within the official PR tests. From an earlier reply, I suppose that this happens because the user does not have the same stale cache that is used by the PR integration tests.

If no code changes will be implemented, may I suggest transferring the files used by the integration tests back to T2_CH_CERN using a PhEDEx user category that is not auto-managed to avoid being removed by the dynamic data placement algorithm?

[schneiml]

@christopheralanwest as [1] shows, there is some ibeos magic involved with the PR test. This explains why there are files in the PR tests.

[1] https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-22262/26154/runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/cmdLog (ctrl-f 'ibeos')

[wmtford]

With the --ibeos option I can successfully run runTheMatrix.py as a user on lxplus. On other cites (cmslpc at FNAL, or my usual T3_US_COLORADO), the dasquery step works, but then step2 fails, presumably because the ibeos environment isn't compatible with these non-CERN sites:
ERROR executing cd 10808.0_SingleMuPt100+SingleMuPt100_pythia8_2018_GenSimFullINPUT+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018; cmsDriver.py step2 --conditions auto:phase1_2018_realistic -s DIGI:pdigi_valid,L1,DIGI2RAW,HLT:@relval2018 --datatier GEN-SIM-DIGI-RAW -n 10 --geometry DB:Extended --era Run2_2018 --eventcontent FEVTDEBUGHLT --number=10 --filein filelist:step1_dasquery.log --fileout file:step2.root > step2_SingleMuPt100+SingleMuPt100_pythia8_2018_GenSimFullINPUT+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018.log 2>&1; ret= 16896

[smuzaffar]

@wmtford , can you please provide the log file of the failed tests with --ibeos?

[wmtford]

Yes, here is the output of runTheMatrix.py:
matrix_ibeos.txt
The first 8 lines come from the batch system at T3_US_COLORADO.
Files generated by the script:
step1_dasquery.log

step2_DIGI_L1_DIGI2RAW_HLT.py.txt

step2_SingleMuPt100+SingleMuPt100_pythia8_2018_GenSimFullINPUT+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018.log

[mmusich]

@smuzaffar and @wmtford
was there perhaps any conclusion on this issue?

@carolinecollard for your information.

[smuzaffar]

@prebello , T2_CH_CERN restriction was added to make sure that we transfer dataset/blocks to CERN under ib-relval group. This is not needed any more as now IBs/PR use files from ibeos area. I have no objections on dropping this restriction or adding a command-line option to drop this.

Let me know which option whould you prefer (adding new command-line option is safe though)?

[mmusich]

I think @prebello left PdmV since some time.
I am adding the current PdmV team instead: @chayanit, @wajidalikhan, @jordan-martins

[chayanit]

Thanks for pointing this to us. Adding new command-line would be a good option. Could you provide the PR? @smuzaffar

[smuzaffar]

#31535 adds --sites <site> option to select a specific site. Setting it to emptry string will search all sites. Default is T2_CH_CERN