CMS: Run2 Hbb and QCD MC for ML studies

[katilp]

In connection with #2440, this issue follows the Hbb-tagging sample to be produced from run2 AOD samples, to be made available on the portal (contact @pierinim @jmduarte)

The datasets:

• signal MC:
  • /BulkGravTohhTohbbhbb_narrow_M-600_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6_ext1-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-1000_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6_ext1-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-1200_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-1400_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-1600_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-1800_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-2000_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-2000_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6_ext1-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-2500_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-3000_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6_ext1-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-4000_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /BulkGravTohhTohbbhbb_narrow_M-4500_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
• qcd bkg
  • /QCD_Pt_300to470_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /QCD_Pt_470to600_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /QCD_Pt_600to800_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /QCD_Pt_800to1000_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /QCD_Pt_1000to1400_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /QCD_Pt_1400to1800_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /QCD_Pt_1800to2400_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /QCD_Pt_2400to3200_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM
    /QCD_Pt_3200toInf_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v3/MINIAODSIM
  • Release: CMSSW_8_0_21 Global Tag: 80X_mcRun2_asymptotic_2016_TrancheIV_v6

To do:

• transfer the MINIAODSIM dataset to T3_CH_CERN_OpenData
  • transfer requests done:
    Hbb ok, QCD ok, both moved to eospublic/cms see CMS: list of datasest to be moved from eospublic/upload to eospublic/cms #2454 for the location
• prepare the data records for MiniAODSIM see CMS: prepare data records for Run2 Hbb and QCD MC for ML studies #2525
• test that the production workflow works on CMS Open Data VM
  • the VM with encapsulated slc6 CMS: new VM for 2011-2012 with slc6 container #2426 can be used (it is not yet on the portal record)
  • the corresponding GT is needed (CMS: GTs for the data science samples #2443)
• test that the production workflow works on ReANA (@alintulu)?
• prepare the CODP record for the production workflow
  • have it in a github repo (see with @caredg ): https://github.com/cms-legacydata-analyses/HiggsToBBNtupleProducerTool
  • provide the corresponding metadata for the CODP record (see CMS: SW record for ML sample production (Run2 Hbb and QCD MC for ML studies) #2585)
• produce the ML files and transfer to eospublic upload (see with @tiborsimko)
  • files: see CMS: Run2 Hbb and QCD MC for ML studies #2448 (comment)
• provide the metadata for the CODP records for the ML files (see CMS: Record for ML samples (Run2 Hbb and QCD MC for ML studies) #2574)
• if appropriate, provide an example of usage as a software record

For contributions, see also https://github.com/cernopendata/opendata.cern.ch/wiki/Contributing-content-to-CERN-Open-Data

[jmduarte]

Hi @katilp, thanks for helping us with this request! Is there an idea of the timeline for these to be available? How can we help with the to-do items?

Thanks!
@pierinim @vlimant @eric-moreno

[katilp]

@jmduarte The files have been transferred, @tiborsimko will move them from the upload area to their final position #2454 and then the production step can be tested on the Open Data VM
It would be good to test with the new version with an embedded "CMS shell" with slc6 #2426 which we will make available at the next release.

The GT 80X_mcRun2_asymptotic_2016_TrancheIV_v8 is now available in /cvmfs/cms-opendata-conddb.cern.ch #2443 and can be read from the Open Data VM in a similar way as in the instructions in http://opendata.cern.ch/docs/cms-guide-for-condition-database
The reading of that GT has not been tested yet so it would be very good if you could try if it works.

[katilp]

@jmduarte The files are now available in /eospublic/cms see #2454

[jmduarte]

@katilp thanks for the update! I'm happy to test the VM. When I try to launch the VM in #2426, and follow the instructions from our ntuplizer (https://github.com/DeepDoubleB/DNNTuplesAK8/README.md):

cmsrel CMSSW_8_0_28
cd CMSSW_8_0_28/src/
cmsenv
git clone https://github.com/DeepDoubleB/DNNTuplesAK8 DeepNTuples -b minpt95_80X
scram b 

I get a warning that I'm on SLC7 and then it fails:

SCRAM warning: You are trying to compile/build for architecture slc6_amd64_gcc530 on SLC7 OS which might not work.
If you know this SCRAM_ARCH/OS combination works then please first run 'scram build --ignore-arch'.

Should I do something different?

Let me know and thanks,
Javier

[katilp]

@jmduarte Note that there are two different shells, the one that opens from the bottom icon is SLC7 and the one from the desktop ("CMS shell") is the SLC6 one. We'll document that better!

[katilp]

Note also #2447 (comment)

[jmduarte]

hi @katilp,

Thanks, I've now verified that this workflow works on the CMS Virtual Machine you linked, following these instructions:
https://github.com/DeepDoubleB/DNNTuplesAK8/blob/opendata_80X/README.md

Note I had to change the way the GlobalTag was read (and adding in a dummy snapshot time too):
https://github.com/DeepDoubleB/DNNTuplesAK8/blob/opendata_80X/NtupleAK8/test/DeepNtuplizerAK8.py#L79-L81

Javier

[katilp]

@jmduarte Thanks, excellent! Did you also try that it runs as a single job on the VM itself without CRAB (just to confirm in order to check the second box in the to-do list above)?
That's needed for the external users who do not have access to CRAB.

[jmduarte]

Hi Kati,

Yes, I just ran a single test job on 100 Hbb events (from the open data EOS space) on the VM (without crab) like this:

cd DeepNTuples/NtupleAK8/test/
cmsRun DeepNtuplizerAK8.py

[katilp]

The ML files are in

root://cmseos.fnal.gov//eos/uscms/store/group/lpcbtag/20181121_ak8_80x/merged_max3files/train/ntuple_merged_*.root
root://cmseos.fnal.gov//eos/uscms/store/group/lpcbtag/20181121_ak8_80x/merged_max3files/test/ntuple_merged_*.root
root://cmseos.fnal.gov//eos/uscms/store/group/lpcbtag/20181121_ak8_80x/merged_max3files/train/ntuple_merged_*.h5
root://cmseos.fnal.gov//eos/uscms/store/group/lpcbtag/20181121_ak8_80x/merged_max3files/test/ntuple_merged_*.h5

[katilp]

Closing as all points addressed