New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Core] Split Eras.py into multiple files #14895
Comments
A new Issue was created by @kpedro88 Kevin Pedro. @davidlange6, @smuzaffar, @Degano, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks. cms-bot commands are listed here |
Currently, a single edit to Eras.py causes checkdeps to suggest 108 dependent packages. Could this issue at least be assigned to someone? |
was thinking about solutions. One can move the ModifiedChain declarations each into a separate file. self.Run2_25ns = cms.ModifierChain( self.run2_common, self.run2_25ns_specific, self.stage1L1Trigger ) becomes from Configuration.Eras.EraRun2_25ns import Run2_25ns where EraRun2_25ns.py is just import FWCore.ParameterSet.Config as cms Run2_25ns = cms.ModifierChain( run2_common, run2_25ns_specific, stage1L1Trigger ) this is easy to implement. The question is then if I change EraRun2_25ns.py does that trigger a massive checkdeps. Shahzad can you comment? I don’t see how to avoid that the list of all eras ends up in one spot. On Aug 5, 2016, at 10:56 PM, Kevin Pedro <notifications@github.commailto:notifications@github.com> wrote: Currently, a single edit to Eras.py causes checkdeps to suggest 108 dependent packages. Could this issue at least be assigned to someone? @davidlange6https://github.com/davidlange6, @smuzaffarhttps://github.com/smuzaffar, @deganohttps://github.com/degano, @davidlthttps://github.com/davidlt, @Dr15Joneshttps://github.com/Dr15Jones — |
@davidlange6, yes checkdeps is still going to checkout all these 108 package (changing EraRun2_25ns.py means Eras.py needs to be re-checked which means every thing depending on Eras.py need to be re-checked) |
ok, thanks - so we need kind of the reverse - a global eras object that then gets used in many places to define the eras - let me try |
thinking more I see two ways - any preferences?
validEras={} class Eras (object):
eras=Eras() followed by: import FWCore.ParameterSet.Config as cms from EraRun2_common import run2_common run2_25ns = cms.ModifierChain( run2_common, run2_25ns_specific, stage1L1Trigger ) etc etc On Aug 8, 2016, at 1:17 PM, Malik Shahzad Muzaffar <notifications@github.commailto:notifications@github.com> wrote: @davidlange6https://github.com/davidlange6, yes checkdeps is still going to checkout all these 108 package (changing EraRun2_25ns.py means Eras.py needs to be re-checked which means every thing depending on Eras.py need to be re-checked) — |
Wouldn't an easier fix be to not have the different |
On Aug 8, 2016, at 3:59 PM, Chris Jones <notifications@github.commailto:notifications@github.com> wrote: Wouldn't an easier fix be to not have the different _cff.py files import the top level Eras module and instead only import a module containing the ProcessModifier they actually use? Then the top level Eras is only used when constructing the Process and by cmsDriver.py to get the available list. its even optional to have that top level Eras file for cmsDriver - it can import the “right” one too (or not?) - maybe its not that painful to script the replacement of eras. by an import of a specific one (which still needs to be in N files or otherwise hidden from the python dependencies check) — |
some work in this direction (and my suggested next steps) On Aug 8, 2016, at 4:06 PM, David Lange <David.Lange@cern.chmailto:David.Lange@cern.ch> wrote: On Aug 8, 2016, at 3:59 PM, Chris Jones <notifications@github.commailto:notifications@github.com> wrote: Wouldn't an easier fix be to not have the different _cff.py files import the top level Eras module and instead only import a module containing the ProcessModifier they actually use? Then the top level Eras is only used when constructing the Process and by cmsDriver.py to get the available list. its even optional to have that top level Eras file for cmsDriver - it can import the “right” one too (or not?) - maybe its not that painful to script the replacement of eras. by an import of a specific one (which still needs to be in N files or otherwise hidden from the python dependencies check) — |
@davidlange6 I think #15480 is sufficient for this issue. Making all the imports more specific is a good followup, but we probably don't need to leave this open until that is done. |
After #16001 and #16086, most importing of
Further investigation with grep:
I've addressed the few leftover tracking, L1, and validation configs in #16357. The DQM and HLT imports appear to be for the purposes of initializing a process, so they should be left alone.
|
I updated Configuration/DataProcessing/python/Impl files in On 10/25/16 2:10 PM, Kevin Pedro wrote:
|
In
@smuzaffar, we should keep an eye on this wrt any fixes based on the discussion in #16385. Otherwise, unless @davidlange6 has further plans for handling Eras python files, we can plan to close this issue soon. |
Checked up on this again and noticed a few more had crept in; fixed in #19732. @davidlange6 @smuzaffar is there some way we could create a unit test to enforce this rule? (i.e. dependence on |
even these dependencies should get removed...that would be more productive use of time that making a unit test imo...
… On Jul 13, 2017, at 3:38 PM, Kevin Pedro ***@***.***> wrote:
Checked up on this again and noticed a few more had crept in; fixed in #19732.
@davidlange6 @smuzaffar is there some way we could create a unit test to enforce this rule? (i.e. dependence on Configuration.StandardSequences.Eras only allowed in the python folders of the three packages listed in #14895 (comment), with the possibility to add other exceptions in the future)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I think the dependence in Even then, we still need to prevent future dependencies from being introduced... |
On Jul 13, 2017, at 3:53 PM, Kevin Pedro ***@***.***> wrote:
I think the dependence in Configuration/Applications is unavoidable (cmsDriver internals). The others are due to full cmsRun configs being tracked in python folders. Do you think we should change cmsDriver so it doesn't create configs that import Configuration.StandardSequences.Eras?
nah - you know what era to call when cmsDriver is started..no need to do anything more than import the one you want. But, right its not implemented.
… Even then, we still need to prevent future dependencies from being introduced...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
@davidlange6 what do you think of this?
If this seems acceptable, I'll first make a PR that goes through all tracked cmsRun configs that import |
hi @kpedro88 - i think getting rid of the Eras_cff altogether in cmsDriver just means something like this. (I didn't look at the other users) |
@davidlange6 I don't think that's general enough. In some cases, the user may want to specify an "internal use modifier", not just a top-level Era. So we need to know how to import any modifier/era. |
@kpedro88 - its an 'era' option not a 'modifier' option - do we have a use case? |
A couple of examples (remember, so far this has been allowed by cmsDriver):
|
@davidlange6 further thoughts on this? |
@davidlange6 ping |
@davidlange6 if this is going to go in 930, I need feedback ASAP |
@kpedro88 , can we close this? |
@smuzaffar I kept meaning to come back to this. I just submitted #26590, which is another step along the path outlined in this issue. |
Can we close this now? Or have a list of steps that would need to be done before it could be closed? |
A trivial modification to
Given that we now also have modifiers in cmssw/Configuration/Applications/python/ConfigBuilder.py Lines 2128 to 2134 in 5cc0e67
cmssw/Configuration/Applications/python/ConfigBuilder.py Lines 2137 to 2145 in 5cc0e67
Eras.py entirely. This would imply the following:
|
We constantly have to deal with collisions in https://github.com/cms-sw/cmssw/blob/CMSSW_8_1_X/Configuration/StandardSequences/python/Eras.py, because all the different development efforts (Run2, Phase1, Phase2) have to modify this file often.
Ideally, we could at least split the Era definitions into a few separate files to reduce pointless collisions (and reduce the volume packages selected by checkdeps when recompiling). However, this requires some redesign, since right now Eras is just a class with a bunch of members.
@davidlange6, @slava77, @Dr15Jones
The text was updated successfully, but these errors were encountered: