Skip to content
This repository was archived by the owner on Sep 12, 2024. It is now read-only.

Rejector update#576

Merged
sharad1126 merged 3 commits intomasterfrom
rejector_changes
Jun 11, 2020
Merged

Rejector update#576
sharad1126 merged 3 commits intomasterfrom
rejector_changes

Conversation

@drkovalskyi
Copy link
Copy Markdown
Collaborator

@drkovalskyi drkovalskyi commented Jun 5, 2020

Status

ready

Description

  • Add a test mode to check schema changes without committing any changes.
  • Enforce single-core for StepChain conversions if a corresponding task in TaskChain is single-core

Is it backward compatible (if not, which system it affects?)

YES

Mention people to look at PRs

@sharad1126, @amaltaro, @z4027163

@drkovalskyi drkovalskyi requested a review from sharad1126 June 8, 2020 17:48
sharad1126
sharad1126 reviewed Jun 9, 2020
View reviewed changes
Comment thread Unified/rejector.py Outdated
Comment thread Unified/rejector.py
Comment thread Unified/rejector.py
@sharad1126
Copy link
Copy Markdown
Contributor

sharad1126 commented Jun 11, 2020

@drkovalskyi I think it looks good to me. I would just try to use the modified script with the --test option once and then merge it.

@sharad1126
Copy link
Copy Markdown
Contributor

sharad1126 commented Jun 11, 2020 •
edited
Loading

I tried the following already rejected request with a --test and -c option. @drkovalskyi I hope you would have expected the same output.

[sagarwal@vocms0275 WmAgentScripts]$ python Unified/rejector.py --test -c pdmvserv_task_EXO-RunIIFall17wmLHEGS-04524__v1_T_200524_013057_9118
Using the rw account
Test mode - no changes propagate to the production system
1 to reject
invalidating the workflow and outputs by unified operator sagarwal
pdmvserv_task_EXO-RunIIFall17wmLHEGS-04524__v1_T_200524_013057_9118 rejected
Original schema
{
  "PrepID": "task_EXO-RunIIFall17wmLHEGS-04524", 
  "DeleteFromSource": false, 
  "MaxMergeEvents": 100000000, 
  "OpenRunningTimeout": 0, 
  "Comments": "", 
  "RobustMerge": true, 
  "Requestor": "pdmvserv", 
  "ProcessingString": "93X_mc2017_realistic_v3", 
  "Multicore": 1, 
  "SizePerEvent": 100, 
  "Task3": {
    "KeepOutput": true, 
    "MCPileup": "/Neutrino_E-10_gun/RunIISummer17PrePremix-MCv2_correctPU_94X_mc2017_realistic_v9-v1/GEN-SIM-DIGI-RAW", 
    "PrepID": "EXO-RunIIFall17DRPremix-05551", 
    "InputFromOutputModule": "PREMIXRAWoutput", 
    "GlobalTag": "94X_mc2017_realistic_v11", 
    "TimePerEvent": 2.0704, 
    "FilterEfficiency": 1, 
    "SplittingAlgo": "EventAwareLumiBased", 
    "ProcessingString": "PU2017_94X_mc2017_realistic_v11", 
    "ScramArch": [
      "slc6_amd64_gcc630"
    ], 
    "SizePerEvent": 700, 
    "ConfigCacheID": "44484f03b56853584f4a8e2c5403bf0c", 
    "Memory": 14700, 
    "Multicore": 8, 
    "TaskName": "EXO-RunIIFall17DRPremix-05551_1", 
    "AcquisitionEra": "RunIIFall17DRPremix", 
    "PrimaryDataset": "ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8", 
    "Campaign": "RunIIFall17DRPremix", 
    "InputTask": "EXO-RunIIFall17DRPremix-05551_0", 
    "CMSSWVersion": "CMSSW_9_4_7"
  }, 
  "Memory": 5900, 
  "IgnoredOutputModules": [], 
  "Group": "ppd", 
  "ChainParentageMap": {
    "Task1": {
      "ParentDset": null, 
      "ChildDsets": []
    }, 
    "Task2": {
      "ParentDset": null, 
      "ChildDsets": []
    }, 
    "Task3": {
      "ParentDset": null, 
      "ChildDsets": [
        "/ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8/RunIIFall17DRPremix-PU2017_94X_mc2017_realistic_v11-v1/AODSIM"
      ]
    }, 
    "Task4": {
      "ParentDset": "/ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8/RunIIFall17DRPremix-PU2017_94X_mc2017_realistic_v11-v1/AODSIM", 
      "ChildDsets": [
        "/ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8/RunIIFall17MiniAODv2-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/MINIAODSIM"
      ]
    }
  }, 
  "DQMUploadUrl": "https://cmsweb.cern.ch/dqm/dev", 
  "Task1": {
    "PrepID": "EXO-RunIIFall17wmLHEGS-04524", 
    "ProcessingString": "93X_mc2017_realistic_v3", 
    "ScramArch": [
      "slc6_amd64_gcc630"
    ], 
    "SizePerEvent": 683, 
    "ConfigCacheID": "d5b7a00146749fb7265af403cf0860a1", 
    "Memory": 4000, 
    "TimePerEvent": 5.958552310130731, 
    "AcquisitionEra": "RunIIFall17wmLHEGS", 
    "PrimaryDataset": "ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8", 
    "CMSSWVersion": "CMSSW_9_3_18", 
    "SplittingAlgo": "EventBased", 
    "LheInputFiles": false, 
    "TaskName": "EXO-RunIIFall17wmLHEGS-04524_0", 
    "KeepOutput": false, 
    "Campaign": "RunIIFall17wmLHEGS", 
    "GlobalTag": "93X_mc2017_realistic_v3", 
    "Seeding": "AutomaticSeeding", 
    "FilterEfficiency": 0.7, 
    "Multicore": 8, 
    "EventsPerLumi": 1428, 
    "EventsPerJob": 4284, 
    "RequestNumEvents": 40000
  }, 
  "Task2": {
    "KeepOutput": false, 
    "MCPileup": "/Neutrino_E-10_gun/RunIISummer17PrePremix-MCv2_correctPU_94X_mc2017_realistic_v9-v1/GEN-SIM-DIGI-RAW", 
    "PrepID": "EXO-RunIIFall17DRPremix-05551", 
    "InputFromOutputModule": "RAWSIMoutput", 
    "GlobalTag": "94X_mc2017_realistic_v11", 
    "TimePerEvent": 1.2181, 
    "FilterEfficiency": 1, 
    "SplittingAlgo": "EventAwareLumiBased", 
    "ProcessingString": "PU2017_94X_mc2017_realistic_v11", 
    "ScramArch": [
      "slc6_amd64_gcc630"
    ], 
    "SizePerEvent": 2341, 
    "ConfigCacheID": "44484f03b56853584f4a8e2c5403bab7", 
    "Memory": 14700, 
    "Multicore": 8, 
    "TaskName": "EXO-RunIIFall17DRPremix-05551_0", 
    "AcquisitionEra": "RunIIFall17DRPremix", 
    "PrimaryDataset": "ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8", 
    "Campaign": "RunIIFall17DRPremix", 
    "InputTask": "EXO-RunIIFall17wmLHEGS-04524_0", 
    "CMSSWVersion": "CMSSW_9_4_7"
  }, 
  "SubRequestType": "ReDigi", 
  "Task4": {
    "KeepOutput": true, 
    "PrepID": "EXO-RunIIFall17MiniAODv2-05491", 
    "InputFromOutputModule": "AODSIMoutput", 
    "GlobalTag": "94X_mc2017_realistic_v14", 
    "TimePerEvent": 1.2, 
    "FilterEfficiency": 1, 
    "SplittingAlgo": "EventAwareLumiBased", 
    "ProcessingString": "PU2017_12Apr2018_94X_mc2017_realistic_v14", 
    "ScramArch": [
      "slc6_amd64_gcc630"
    ], 
    "SizePerEvent": 100, 
    "ConfigCacheID": "44484f03b56853584f4a8e2c5403e300", 
    "Memory": 5900, 
    "Multicore": 4, 
    "TaskName": "EXO-RunIIFall17MiniAODv2-05491_0", 
    "AcquisitionEra": "RunIIFall17MiniAODv2", 
    "PrimaryDataset": "ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8", 
    "Campaign": "RunIIFall17MiniAODv2", 
    "InputTask": "EXO-RunIIFall17DRPremix-05551_1", 
    "CMSSWVersion": "CMSSW_9_4_7"
  }, 
  "DashboardPort": 8884, 
  "MergedLFNBase": "/store/data", 
  "TimePerEvent": 1.2, 
  "LumiList": {}, 
  "DQMSequences": [], 
  "OutputDatasets": [
    "/ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8/RunIIFall17DRPremix-PU2017_94X_mc2017_realistic_v11-v1/AODSIM", 
    "/ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8/RunIIFall17MiniAODv2-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/MINIAODSIM"
  ], 
  "VoRole": "unknown", 
  "DQMHarvestUnit": "byRun", 
  "PriorityTransition": [
    {
      "Priority": 80000, 
      "DN": "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch", 
      "UpdateTime": 1590276657
    }, 
    {
      "DN": "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch", 
      "Priority": 80001, 
      "UpdateTime": 1591046803
    }
  ], 
  "EnableNewStageout": false, 
  "DashboardHost": "cms-jobmon.cern.ch", 
  "EnableHarvesting": false, 
  "ProcessingVersion": 1, 
  "IncludeParents": false, 
  "RequestPriority": 80001, 
  "MaxMergeSize": 4294967296, 
  "PeriodicHarvestInterval": 0, 
  "_id": "pdmvserv_task_EXO-RunIIFall17wmLHEGS-04524__v1_T_200524_013057_9118", 
  "MaxWaitTime": 86400, 
  "UnmergedLFNBase": "/store/unmerged", 
  "FirstEvent": 1, 
  "RequestName": "pdmvserv_task_EXO-RunIIFall17wmLHEGS-04524__v1_T_200524_013057_9118", 
  "MinMergeSize": 2147483648, 
  "RequestString": "task_EXO-RunIIFall17wmLHEGS-04524__v1_T", 
  "CouchURL": "https://cmsweb.cern.ch/couchdb", 
  "ValidStatus": "PRODUCTION", 
  "CouchWorkloadDBName": "reqmgr_workload_cache", 
  "RequestWorkflow": "https://cmsweb.cern.ch/couchdb/reqmgr_workload_cache/pdmvserv_task_EXO-RunIIFall17wmLHEGS-04524__v1_T_200524_013057_9118/spec", 
  "Campaign": "RunIIFall17wmLHEGS", 
  "GlobalTag": "94X_mc2017_realistic_v14", 
  "TaskChain": 4, 
  "FirstLumi": 1, 
  "InitialPriority": 80000, 
  "RequestorDN": "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch", 
  "VoGroup": "unknown", 
  "RequestType": "TaskChain", 
  "RequestTransition": [
    {
      "Status": "new", 
      "DN": "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch", 
      "UpdateTime": 1590276657
    }, 
    {
      "DN": "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch", 
      "Status": "assignment-approved", 
      "UpdateTime": 1591137737
    }, 
    {
      "DN": "/DC=ch/DC=cern/OU=computers/CN=unified/vocms0268.cern.ch", 
      "Status": "rejected", 
      "UpdateTime": 1591331228
    }, 
    {
      "DN": "/DC=ch/DC=cern/OU=computers/CN=dmwm/cmsweb.cern.ch", 
      "Status": "rejected-archived", 
      "UpdateTime": 1591417676
    }
  ], 
  "DbsUrl": "https://cmsweb.cern.ch/dbs/prod/global/DBSReader/", 
  "ScramArch": [
    "slc6_amd64_gcc630"
  ], 
  "RequestDate": [
    2020, 
    5, 
    23, 
    23, 
    30, 
    57
  ], 
  "CouchDBName": "reqmgr_config_cache", 
  "ConfigCacheUrl": "https://cmsweb.cern.ch/couchdb", 
  "CMSSWVersion": "CMSSW_9_4_7", 
  "RequestStatus": "rejected-archived", 
  "AcquisitionEra": "RunIIFall17wmLHEGS", 
  "RunNumber": 0
}
submitting
New request schema
{
  "PrepID": "task_EXO-RunIIFall17wmLHEGS-04524", 
  "DeleteFromSource": false, 
  "OpenRunningTimeout": 0, 
  "Comments": "", 
  "RobustMerge": true, 
  "Requestor": "sagarwal", 
  "ProcessingString": "93X_mc2017_realistic_v3", 
  "Multicore": 1, 
  "SizePerEvent": 100, 
  "Task3": {
    "KeepOutput": true, 
    "MCPileup": "/Neutrino_E-10_gun/RunIISummer17PrePremix-MCv2_correctPU_94X_mc2017_realistic_v9-v1/GEN-SIM-DIGI-RAW", 
    "PrepID": "EXO-RunIIFall17DRPremix-05551", 
    "InputFromOutputModule": "PREMIXRAWoutput", 
    "GlobalTag": "94X_mc2017_realistic_v11", 
    "TimePerEvent": 2.0704, 
    "FilterEfficiency": 1, 
    "SplittingAlgo": "EventAwareLumiBased", 
    "ProcessingString": "PU2017_94X_mc2017_realistic_v11", 
    "ScramArch": [
      "slc6_amd64_gcc630"
    ], 
    "SizePerEvent": 700, 
    "ConfigCacheID": "44484f03b56853584f4a8e2c5403bf0c", 
    "Memory": 14700, 
    "Multicore": 8, 
    "TaskName": "EXO-RunIIFall17DRPremix-05551_1", 
    "AcquisitionEra": "RunIIFall17DRPremix", 
    "PrimaryDataset": "ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8", 
    "Campaign": "RunIIFall17DRPremix", 
    "InputTask": "EXO-RunIIFall17DRPremix-05551_0", 
    "CMSSWVersion": "CMSSW_9_4_7"
  }, 
  "Memory": 5900, 
  "Group": "DATAOPS", 
  "ChainParentageMap": {
    "Task1": {
      "ParentDset": null, 
      "ChildDsets": []
    }, 
    "Task2": {
      "ParentDset": null, 
      "ChildDsets": []
    }, 
    "Task3": {
      "ParentDset": null, 
      "ChildDsets": [
        "/ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8/RunIIFall17DRPremix-PU2017_94X_mc2017_realistic_v11-v1/AODSIM"
      ]
    }, 
    "Task4": {
      "ParentDset": "/ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8/RunIIFall17DRPremix-PU2017_94X_mc2017_realistic_v11-v1/AODSIM", 
      "ChildDsets": [
        "/ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8/RunIIFall17MiniAODv2-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/MINIAODSIM"
      ]
    }
  }, 
  "DQMUploadUrl": "https://cmsweb.cern.ch/dqm/dev", 
  "Task1": {
    "PrepID": "EXO-RunIIFall17wmLHEGS-04524", 
    "ProcessingString": "93X_mc2017_realistic_v3", 
    "ScramArch": [
      "slc6_amd64_gcc630"
    ], 
    "SizePerEvent": 683, 
    "ConfigCacheID": "d5b7a00146749fb7265af403cf0860a1", 
    "Memory": 4000, 
    "TimePerEvent": 5.958552310130731, 
    "AcquisitionEra": "RunIIFall17wmLHEGS", 
    "PrimaryDataset": "ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8", 
    "CMSSWVersion": "CMSSW_9_3_18", 
    "SplittingAlgo": "EventBased", 
    "LheInputFiles": false, 
    "TaskName": "EXO-RunIIFall17wmLHEGS-04524_0", 
    "KeepOutput": false, 
    "Campaign": "RunIIFall17wmLHEGS", 
    "GlobalTag": "93X_mc2017_realistic_v3", 
    "Seeding": "AutomaticSeeding", 
    "FilterEfficiency": 0.7, 
    "Multicore": 8, 
    "EventsPerLumi": 1428, 
    "RequestNumEvents": 40000
  }, 
  "Task2": {
    "KeepOutput": false, 
    "MCPileup": "/Neutrino_E-10_gun/RunIISummer17PrePremix-MCv2_correctPU_94X_mc2017_realistic_v9-v1/GEN-SIM-DIGI-RAW", 
    "PrepID": "EXO-RunIIFall17DRPremix-05551", 
    "InputFromOutputModule": "RAWSIMoutput", 
    "GlobalTag": "94X_mc2017_realistic_v11", 
    "TimePerEvent": 1.2181, 
    "FilterEfficiency": 1, 
    "SplittingAlgo": "EventAwareLumiBased", 
    "ProcessingString": "PU2017_94X_mc2017_realistic_v11", 
    "ScramArch": [
      "slc6_amd64_gcc630"
    ], 
    "SizePerEvent": 2341, 
    "ConfigCacheID": "44484f03b56853584f4a8e2c5403bab7", 
    "Memory": 14700, 
    "Multicore": 8, 
    "TaskName": "EXO-RunIIFall17DRPremix-05551_0", 
    "AcquisitionEra": "RunIIFall17DRPremix", 
    "PrimaryDataset": "ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8", 
    "Campaign": "RunIIFall17DRPremix", 
    "InputTask": "EXO-RunIIFall17wmLHEGS-04524_0", 
    "CMSSWVersion": "CMSSW_9_4_7"
  }, 
  "SubRequestType": "ReDigi", 
  "Task4": {
    "KeepOutput": true, 
    "PrepID": "EXO-RunIIFall17MiniAODv2-05491", 
    "InputFromOutputModule": "AODSIMoutput", 
    "GlobalTag": "94X_mc2017_realistic_v14", 
    "TimePerEvent": 1.2, 
    "FilterEfficiency": 1, 
    "SplittingAlgo": "EventAwareLumiBased", 
    "ProcessingString": "PU2017_12Apr2018_94X_mc2017_realistic_v14", 
    "ScramArch": [
      "slc6_amd64_gcc630"
    ], 
    "SizePerEvent": 100, 
    "ConfigCacheID": "44484f03b56853584f4a8e2c5403e300", 
    "Memory": 5900, 
    "Multicore": 4, 
    "TaskName": "EXO-RunIIFall17MiniAODv2-05491_0", 
    "AcquisitionEra": "RunIIFall17MiniAODv2", 
    "PrimaryDataset": "ggH_HToSSTo4l_MH-800_MS-350_ctauS-1000_TuneCP5_13TeV-powheg-pythia8", 
    "Campaign": "RunIIFall17MiniAODv2", 
    "InputTask": "EXO-RunIIFall17DRPremix-05551_1", 
    "CMSSWVersion": "CMSSW_9_4_7"
  }, 
  "DashboardPort": 8884, 
  "MergedLFNBase": "/store/data", 
  "TimePerEvent": 1.2, 
  "LumiList": {}, 
  "DQMSequences": [], 
  "VoRole": "unknown", 
  "DQMHarvestUnit": "byRun", 
  "PriorityTransition": [
    {
      "Priority": 80000, 
      "DN": "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch", 
      "UpdateTime": 1590276657
    }, 
    {
      "DN": "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch", 
      "Priority": 80001, 
      "UpdateTime": 1591046803
    }
  ], 
  "EnableNewStageout": false, 
  "DashboardHost": "cms-jobmon.cern.ch", 
  "EnableHarvesting": false, 
  "ProcessingVersion": 2, 
  "IncludeParents": false, 
  "RequestPriority": 80001, 
  "PeriodicHarvestInterval": 0, 
  "MaxWaitTime": 86400, 
  "UnmergedLFNBase": "/store/unmerged", 
  "FirstEvent": 1, 
  "RequestString": "task_EXO-RunIIFall17wmLHEGS-04524__v1_T", 
  "CouchURL": "https://cmsweb.cern.ch/couchdb", 
  "ValidStatus": "PRODUCTION", 
  "Campaign": "RunIIFall17wmLHEGS", 
  "GlobalTag": "94X_mc2017_realistic_v14", 
  "TaskChain": 4, 
  "FirstLumi": 1, 
  "RequestorDN": "/DC=ch/DC=cern/OU=computers/CN=pdmvserv/vocms081.cern.ch", 
  "VoGroup": "unknown", 
  "RequestType": "TaskChain", 
  "DbsUrl": "https://cmsweb.cern.ch/dbs/prod/global/DBSReader/", 
  "ScramArch": [
    "slc6_amd64_gcc630"
  ], 
  "RequestDate": [
    2020, 
    5, 
    23, 
    23, 
    30, 
    57
  ], 
  "CouchDBName": "reqmgr_config_cache", 
  "ConfigCacheUrl": "https://cmsweb.cern.ch/couchdb", 
  "CMSSWVersion": "CMSSW_9_4_7", 
  "AcquisitionEra": "RunIIFall17wmLHEGS", 
  "RunNumber": 0
}

@sharad1126 sharad1126 merged commit 46c364f into master Jun 11, 2020
@drkovalskyi
Copy link
Copy Markdown
Collaborator Author

drkovalskyi commented Jun 11, 2020

The output looks reasonable. I'm using this branch for all recent production clones that I made.

@sharad1126 sharad1126 deleted the rejector_changes branch August 26, 2020 14:13
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@sharad1126 sharad1126 sharad1126 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@drkovalskyi @sharad1126