Add additional validation samples

[jpata]

So far we were testing with a FlatQCD sample. Now need to add a few additional ones (PU & no PU, Zmm, MinBias). One issue is making sure we can access these samples from CERN, even if they are stored at FNAL.

[jpata]

I looked into it a little bit and at CERN we only have the 10_4_X samples, as that is the dev release.
Issue #21.

[jpata]

I'm having issues with the dataset availability at CERN, I've made a hypernews thread here: https://hypernews.cern.ch/HyperNews/CMS/get/comp-ops/4388.html

[jpata]

I was not able to get a response on transferring datasets to CERN. Not sure who is managing T2_CH_CERN and under what conditions can we transfer data there using Phedex.
In any case, datasets can be defined here: https://github.com/jpata/cmssw/blob/pfvalidation-10_4_X-correct/Validation/RecoParticleFlow/test/datasets.py#L129

[jpata]

OK, so we can't ask to transfer those datasets to CERN (see hn thread). I'm currently seeing how reliable reading them via xrootd from Caltech batch jobs will be.
Added ZMM and MinBias samples here: d9a89a7

[jpata]

I discussed with Jean-Roch Vlimant, and to us it seemed more logical to transfer 1-2TB of samples to CERN than rely on xrootd daily in hundreds of batch jobs. Indeed, 99% of my batch jobs on lxplus failed because of the xrootd connection from CERN to FNAL.

Here is the Phedex request for the samples: https://cmsweb.cern.ch/phedex/prod/Request::View?request=1620067

[juska]

Excellent! I can see that the following data are being moved to CERN T2: /RelValMinBias_13/CMSSW_10_4_0_pre4-103X_mc2017_realistic_v2-v1/GEN-SIM-DIGI-RAW /RelValQCD_FlatPt_15_3000HS_13/CMSSW_10_4_0_pre4-103X_mc2017_realistic_v2-v1/GEN-SIM-DIGI-RAW /RelValQCD_FlatPt_15_3000HS_13/CMSSW_10_4_0_pre4-PU25ns_103X_mc2017_realistic_v2-v1/GEN-SIM-DIGI-RAW /RelValZMM_13/CMSSW_10_4_0_pre4-103X_mc2017_realistic_v2-v1/GEN-SIM-DIGI-RAW What about any new RelVal data, will we have to keep on doing these requests constantly? For physics datasets it's possible to do a phedex transfer request that will top up the destination with new data as soon as it becomes available, do you know if something like this is possible for RelVal data 'tiers'? I mean that we would 'subscribe' to /RelValMinBias_13, /RelValQCD_FlatPt_15_3000HS_13, /RelValZMM_13, and what ever datasets we see needed in the future. Cheers, Juska

…

On 2/6/19 7:52 PM, Joosep Pata wrote: I discussed with Jean-Roch Vlimant, and to us it seemed more logical to transfer 1-2TB of samples to CERN than rely on xrootd daily in hundreds of batch jobs. Indeed, 99% of my batch jobs on lxplus failed because of the xrootd connection from CERN to FNAL. Here is the Phedex request for the samples: https://cmsweb.cern.ch/phedex/prod/Request::View?request=1620067 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHKcoAqjPlyDgRa3UXAdy8dovAuzUAFbks5vKyR2gaJpZM4aa0L->.

[jpata]

To close this topic, we will likely have to ask for data to be moved to CERN if we want to run the code only at CERN. Likely we will need one set of datasets per major CMSSW release (few TB), not every RelVal (many TB per release?). LXbatch condor jobs opening files at FNAL will work but require resubmission occasionally.