CMS: provenance metadata for 2015 MC

[katilp]

(from #65)

Run2 legacy metadata of 2016-2018 will be extracted from the ultra-legacy production, see https://github.com/cernopendata/data-curation/tree/master/cms-run2-ultra-legacy-production.

Run1 metadata has been extracted with

• https://github.com/cernopendata/data-curation/tree/master/cms-2010-simulated-datasets
• https://github.com/cernopendata/data-curation/tree/master/cms-2011-simulated-datasets
• https://github.com/cernopendata/data-curation/tree/master/cms-2012-simulated-datasets

2015 is a separate production and the script may need to be modified.

Run2 Fall15 MiniAOD v2 campaign (76X version 2)dataset=/*/*RunIIFall15MiniAODv2-PU25nsData2015v1*/*
Size: 227.4TB
7114 datasets

Check if the 2015 MC extraction works with the ultra-legacy script, or if the previous Run1 scripts should be used.

Run the script and prepare the records.

[katilp]

NB that for Run1, the datasets are in AODSIM format, but for Run2 they will be in MiniAODSIM, which is an additional step in the production chain, which was not present during Run1.
See an example of a MiniAODSIM dataset that has been released in the context of ML sample release July 2019. The outcome of the script should be like that.

[katilp]

Because of the features in 2015 MC production, some McM entries do not have the input data field filled although it is available in DAS.

The script in https://github.com/cernopendata/data-curation/tree/master/cms-YYYY-simulated-datasets can be used for all further provenance extractions. The query logic (parent relationship from DAS), should not be modified for 2015, but can be done for UL afterwards.

To test:

• Make a short CMS-0000-mc-datasets.txt in input
• Go to bash
• voms-proxy-init --voms cms --rfc --valid 190:00
• Edit script to give no-create-eos-indexes and input to the first step
• go to python3 e.g. with source /afs/cern.ch/user/r/reana/public/reana/bin/activate and check python version with python -V
• Run YYYY=0000 ./make_local_cache.sh

To do:

• comment out the config extraction, and check in how many cases input data is not available in McM if running through all files.

[katilp]

Extraction done by @OsamaMomani
some modifications to the script, all to be added in a new cms-2015-simulated-datasets folder

OK, but provenance not found for three datasets. Manual search through DAS and MCM gives the following:

• /AToZhToLLTauTau_M-260_13TeV_madgraph_4f_LO/RunIIFall15DR76-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/AODSIM
  • das link: only forward config recorded
  • GEN-SIM parent from das:
    • with SIM and HLT configs
    • with the RECO config for AODSIM
    • no entry in McM
  • LHE parent from das:
    • with entry in McM
• /GG_M-2000To4000_Pt-70_13TeV-sherpa/RunIIFall15DR76-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/AODSIM
  • das link flagged as INVALID
• /ZprimeToA0hToA0chichihZZTo4l_2HDM_MZp-1400_MA0-300_13TeV-madgraph-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v2/MINIAODSIM
  • das link
    • no config in das
    • no entry in McM
  • AODSIM parent from das
    • no config in das
    • no parent from das
    • entry in McM with HLT and RECO configs
  • GEN-SIM parent from McM

➡️ Leave out the invalid one, and build the provenance of the two others by hand.

[OsamaMomani]

These 3 datasets affect three steps in the following records

1. For the record /AToZhToLLTauTau_M-260_13TeV_madgraph_4f_LO/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
   • recid: 15333
   • step 3 out of 4
   • step in cms-simulated-datasets-2015-part_1.json lines 19425-19430
2. For the record /GG_M-2000To4000_Pt-70_13TeV-sherpa/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
   • recid 16770
   • step 2 out of 3
   • step in cms-simulated-datasets-2015-part_1.json lines 254912-254917
3. For the record /ZprimeToA0hToA0chichihZZTo4l_2HDM_MZp-1400_MA0-300_13TeV-madgraph-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v2/MINIAODSIM
   • recid 21714
   • step 3 out of 3
   • step in cms-simulated-datasets-2015-part_2.json lines 547417-547422

[katilp]

• https://opendata-qa.cern.ch/record/15333 - step HLT RECO information missing:

  Step
  Release: CMSSW_7_6_1
  Output dataset: /AToZhToLLTauTau_M-260_13TeV_madgraph_4f_LO/RunIIFall15DR76-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/AODSIM
  

  • HLT from DAS: https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/2db80034f6fb21445f516c7c535757aa/configFile
  • RECO from DAS: https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/2db80034f6fb21445f516c7c536caa03/configFile
• https://opendata-qa.cern.ch/record/16770 - step HLT RECO information missing

  Step
  Release: CMSSW_7_6_1
  Output dataset: /GG_M-2000To4000_Pt-70_13TeV-sherpa/RunIIFall15DR76-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/AODSIM
  

  • HLT from DAS: https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/8941377dce285a78c913aa0df0f8481b/configFile
  • RECO from DAS: https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/1afa7fc4132ca1afecdba4b11c0d3b18/configFile
• https://opendata-qa.cern.ch/record/21714

  Step
  Release: CMSSW_7_6_3
  Output dataset: /ZprimeToA0hToA0chichihZZTo4l_2HDM_MZp-1400_MA0-300_13TeV-madgraph-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v2/MINIAODSIM
  

  • MINIAODSIM from DAS: https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/e0d53bd10a30895a034210f18a7b39db/configFile

McM queries are empty for these output datasets.

[katilp]

The missing 5 configs are in /eos/opendata/cms/upload/kati/addedconfigs ready to be move under /eos/opendata/cms/configuration-files/MonteCarlo2015

The records are in

cernopendata/modules/fixtures/data/records/cms-simulated-datasets-2015-part_01.json:    "recid": "15333",
cernopendata/modules/fixtures/data/records/cms-simulated-datasets-2015-part_04.json:    "recid": "16770",
cernopendata/modules/fixtures/data/records/cms-simulated-datasets-2015-part_14.json:    "recid": "21714",

[tiborsimko]

The 5 files are already in the target location, for example:

/eos/opendata/cms/configuration-files/MonteCarlo2015/1afa7fc4132ca1afecdba4b11c0d3b18.configFile.py