Configuration parameter copy and modify functions extended #13224

Dr15Jones · 2016-02-08T19:36:35Z

The clone and Modifier.toModify methods have been extended to allow further modifications to the parameters after they have been copied.

Specify None as the value of the parameter will cause the parameter to be removed from the configuration
Parameters within embedded PSets can now be modified using a python dict as the value. The keys of the dictionary are the parameter
names and the values are the new values to use.
The clone and Modifier.toModify both call the same helper function to do the work of modifying the copied parameters.

The `clone` and `Modifier.toModify` methods have been extended to allow further modifications to the parameters after they have been copied. 1) Specify `None` as the value of the parameter will cause the parameter to be removed from the configuration 2) Parameters within embedded PSets can now be modified using a python dict as the value. The keys of the dictionary are the parameter names and the values are the new values to use. The `clone` and `Modifier.toModify` both call the same helper function to do the work of modifying the copied parameters.

cmsbuild · 2016-02-08T19:36:58Z

A new Pull Request was created by @Dr15Jones (Chris Jones) for CMSSW_8_0_X.

It involves the following packages:

FWCore/ParameterSet

@cmsbuild, @smuzaffar, @Dr15Jones, @davidlange6 can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @wddgit, @wmtan this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

Dr15Jones · 2016-02-08T19:37:05Z

please test

Dr15Jones · 2016-02-08T19:37:10Z

+1

cmsbuild · 2016-02-08T19:37:15Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/11075/console

cmsbuild · 2016-02-08T19:37:15Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_0_X IBs after it passes the integration tests. This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @Degano, @smuzaffar

cmsbuild · 2016-02-08T22:20:51Z

-1
Tested at: f574c22
I found an error when building:

In file included from /afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/external/boost/1.57.0-kpegke/include/boost/smart_ptr/detail/spinlock_pool.hpp:25:
In file included from /afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/external/boost/1.57.0-kpegke/include/boost/smart_ptr/detail/spinlock.hpp:50:
In file included from /afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/external/boost/1.57.0-kpegke/include/boost/smart_ptr/detail/spinlock_sync.hpp:18:
In file included from /afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/external/boost/1.57.0-kpegke/include/boost/smart_ptr/detail/yield_k.hpp:28:
In file included from /afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/external/boost/1.57.0-kpegke/include/boost/predef.h:16:
/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/external/boost/1.57.0-kpegke/include/boost/predef/other.h:11:10: fatal error: cannot open file '/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/external/boost/1.57.0-kpegke/include/boost/predef/other/endian.h': Bad address
#include 
         ^
Error: rootcling: compilation failure (tmp/slc6_amd64_gcc493/src/L1Trigger/L1TGlobal/src/L1TriggerL1TGlobal/a/L1TriggerL1TGlobal_xrb0c432e16c_dictUmbrella.h)
gmake: **\* [tmp/slc6_amd64_gcc493/src/L1Trigger/L1TGlobal/src/L1TriggerL1TGlobal/a/L1TriggerL1TGlobal_xr.cc] Error 2
>> Compiling  /tmp/cmsbuild/workspace/ib-any-integration/CMSSW_8_0_X_2016-02-07-2300/src/L1Trigger/L1TGlobal/src/CaloTemplate.cc

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13224/11075/summary.html

Dr15Jones · 2016-02-09T16:17:49Z

please test

cmsbuild · 2016-02-09T16:18:08Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/11095/console
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/11110/console
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/11111/console

Configuration parameter copy and modify functions extended

Martin-Grunewald · 2016-02-10T07:27:35Z

@Dr15Jones

I am afraid either this PR or #13220 broke things, eg, cmsDriver:

Creating DigiL1Raw GRun_MC
DIGI,L1,DIGI2RAW,ENDJOB
entry /store/relval/CMSSW_7_6_2/RelValProdTTbar_13/GEN-SIM/76X_mcRun2_asymptotic_v12-v1/00000/88639327-209D-E511-BC0D-0025905A6136.root
 ################################################################### 
 # WARNING: this module is deprecated.                             # 
 # Please use CondCore.CondDB.CondDB_cfi.py                        # 
 ################################################################### 
Step: DIGI Spec: 
Step: L1 Spec: 
Step: DIGI2RAW Spec: 
Traceback (most recent call last):
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/bin/slc6_amd64_gcc493/cmsDriver.py", line 55, in <module>
    run()
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/bin/slc6_amd64_gcc493/cmsDriver.py", line 27, in run
    configBuilder.prepare()
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/Configuration/Applications/ConfigBuilder.py", line 2058, in prepare
    self.addStandardSequences()
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/Configuration/Applications/ConfigBuilder.py", line 739, in addStandardSequences
    getattr(self,"prepare_"+stepName)(sequence = getattr(self,stepName+"DefaultSeq"))
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/Configuration/Applications/ConfigBuilder.py", line 1482, in prepare_DIGI2RAW
    self.loadDefaultOrSpecifiedCFF(sequence,self.DIGI2RAWDefaultCFF)
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/Configuration/Applications/ConfigBuilder.py", line 1189, in loadDefaultOrSpecifiedCFF
    l=self.loadAndRemember(defaultCFF,unsch)
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/Configuration/Applications/ConfigBuilder.py", line 323, in loadAndRemember
    self.process.load(includeFile)
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/FWCore/ParameterSet/Config.py", line 511, in load
    module = __import__(moduleName)
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/Configuration/StandardSequences/DigiToRaw_cff.py", line 19, in <module>
    from EventFilter.CSCRawToDigi.cscPacker_cfi import *
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/EventFilter/CSCRawToDigi/cscPacker_cfi.py", line 26, in <module>
    eras.run2_common.toModify( cscpacker, useFormatVersion = cms.uint32(2013) )
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/FWCore/ParameterSet/Config.py", line 1183, in toModify
    temp(obj)
  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/FWCore/ParameterSet/Config.py", line 1112, in __call__
    params[k] = getattr(obj,k)
AttributeError: 'EDProducer' object has no attribute 'useFormatVersion'

Could you please have a look?

smuzaffar · 2016-02-10T07:55:17Z

@Dr15Jones , I think both PR might have broke the IB. At least reverting this PR I am able to run wf 5.1,5.2,5.3 which were failing due to

  File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/FWCore/ParameterSet/Mixins.py", line 634, in _modifyParametersFromDict
    params[key].setValue(value)
AttributeError: '_SequenceCollection' object has no attribute 'setValue'

so we shoudl revert this PR. I will check #13220 too

smuzaffar · 2016-02-10T07:58:59Z

@Dr15Jones , it is this PR which broke the IB. reverting it I can also run 140.2 which was failing due to

File "/afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-02-09-2300/python/FWCore/ParameterSet/Config.py", line 1112, in __call__
    params[k] = getattr(obj,k)
AttributeError: 'EDProducer' object has no attribute 'useFormatVersion'

smuzaffar · 2016-02-10T08:00:16Z

@davidlange6 , we might not get a fix for this before 11h00 IB, so I propose to revert this PR

smuzaffar · 2016-02-10T09:23:19Z

I confirm that #13216 does fix the step1 errors if 5.1,5.2 and all other. 140.2 and other step2 errors are still there

davidlange6 · 2016-02-10T09:26:48Z

ok, could you revert it?

On Feb 10, 2016, at 10:23 AM, Malik Shahzad Muzaffar notifications@github.com wrote:

I confirm that #13216 does fix the step1 errors if 5.1,5.2 and all other. 140.2 and other step2 errors are still there

—
Reply to this email directly or view it on GitHub.

davidlange6 · 2016-02-10T09:30:17Z

err- sorry, I misunderstand
I can merge 13216 if that is what you mean?

On Feb 10, 2016, at 10:26 AM, David Lange David.Lange@cern.ch wrote:

ok, could you revert it?

On Feb 10, 2016, at 10:23 AM, Malik Shahzad Muzaffar notifications@github.com wrote:

I confirm that #13216 does fix the step1 errors if 5.1,5.2 and all other. 140.2 and other step2 errors are still there

—
Reply to this email directly or view it on GitHub.

smuzaffar · 2016-02-10T09:34:53Z

No, even after merging #13216 step2 of many workflows (140.2 and others) will be broken.

cmsbuild · 2016-02-10T11:19:29Z

-1

Tested at: f574c22
I found errors in the following unit tests:

---> test runtestSimCalorimetryHGCalSimProducers had ERRORS
---> test runtestRecoLocalCaloHGCalRecProducers had ERRORS
---> test testRecoMETMETProducers had ERRORS

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13224/11111/summary.html

Dr15Jones · 2016-02-10T13:55:11Z

I know what the problem is with

'EDProducer' object has no attribute 'useFormatVersion'

And can fix it. Matti's other change to use toReplaceWith should solve the other problems.

I'll get a fix for the above this morning (although I'm at home sick so might not be quite as fast as usual).

Dr15Jones · 2016-02-10T17:46:02Z

#13244 includes and fixes this one

cmsbuild added this to the Next CMSSW_8_0_X milestone Feb 8, 2016

cmsbuild added core-pending pending-signatures tests-pending orp-pending comparison-pending labels Feb 8, 2016

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Feb 8, 2016

cmsbuild added tests-started and removed tests-pending labels Feb 8, 2016

This was referenced Feb 8, 2016

Use a dict to specify how to change subvalues of a PSet #13210

Closed

Use 'None' to denote that a parameter should be removed #13209

Closed

cmsbuild added tests-rejected and removed tests-started labels Feb 8, 2016

cmsbuild added tests-pending and removed tests-rejected labels Feb 9, 2016

cmsbuild added tests-started and removed tests-pending labels Feb 9, 2016

davidlange6 added a commit that referenced this pull request Feb 9, 2016

Merge pull request #13224 from Dr15Jones/extendedPSetModification

0dc13c7

Configuration parameter copy and modify functions extended

davidlange6 merged commit 0dc13c7 into cms-sw:CMSSW_8_0_X Feb 9, 2016

makortel mentioned this pull request Feb 10, 2016

Use Modifier.toReplaceWith() in TrackValidation_cff #13216

Merged

smuzaffar mentioned this pull request Feb 10, 2016

Revert "Configuration parameter copy and modify functions extended" #13235

Merged

cmsbuild added tests-rejected and removed tests-started labels Feb 10, 2016

Dr15Jones deleted the extendedPSetModification branch February 15, 2016 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuration parameter copy and modify functions extended #13224

Configuration parameter copy and modify functions extended #13224

Dr15Jones commented Feb 8, 2016

cmsbuild commented Feb 8, 2016

Dr15Jones commented Feb 8, 2016

Dr15Jones commented Feb 8, 2016

cmsbuild commented Feb 8, 2016

cmsbuild commented Feb 8, 2016

cmsbuild commented Feb 8, 2016

Dr15Jones commented Feb 9, 2016

cmsbuild commented Feb 9, 2016

Martin-Grunewald commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

davidlange6 commented Feb 10, 2016

davidlange6 commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

cmsbuild commented Feb 10, 2016

Dr15Jones commented Feb 10, 2016

Dr15Jones commented Feb 10, 2016

Configuration parameter copy and modify functions extended #13224

Configuration parameter copy and modify functions extended #13224

Conversation

Dr15Jones commented Feb 8, 2016

cmsbuild commented Feb 8, 2016

Dr15Jones commented Feb 8, 2016

Dr15Jones commented Feb 8, 2016

cmsbuild commented Feb 8, 2016

cmsbuild commented Feb 8, 2016

cmsbuild commented Feb 8, 2016

Dr15Jones commented Feb 9, 2016

cmsbuild commented Feb 9, 2016

Martin-Grunewald commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

davidlange6 commented Feb 10, 2016

davidlange6 commented Feb 10, 2016

smuzaffar commented Feb 10, 2016

cmsbuild commented Feb 10, 2016

Dr15Jones commented Feb 10, 2016

Dr15Jones commented Feb 10, 2016