Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete Geant4 MT integration #4505

Merged
merged 38 commits into from Jul 5, 2014
Merged

Conversation

makortel
Copy link
Contributor

@makortel makortel commented Jul 3, 2014

This PR completes (almost, see below for a known issue) the integration of Geant4 MT into multithreaded CMSSW. Most modifications are again in MT-specific classes, but some classes used by the serial OscarProducer were also modified.

Changes affecting OscarProducer (results should stay the same)

  • Use G4Random::setTheEngine() instead of CLHEP::HepRandom::setTheEngine()
    • Otherwise incorrect random number engine is used if run in MT G4 build
    • Serial G4 build has #define G4Random CLHEP::HepRandom, so should be ok
  • Fix CustomUIsession to redirect G4cout/G4cerr to LogInfo/LogWarning
  • New class sim::ChordFinderSetter, abstracted out of sim::FieldBuilder, pass it to PhysicsListMakerBase::make()
    • The only reason sim::FieldBuilder was passed to physics list was that G4MonopoleTransporation needs to switch between two G4ChordFinder objects. PhysicsListMakerBase::make() must be called in master thread, and sim::FieldBuilder::build() in all worker threads. This separation was the fastest way forward, althought it required changing the interface of PhysicsListMakerBase::make(), and percolating to all physics list classes.

Changes in OscarMTProducer

  • Merged RunManagerMTInit to OscarMTMasterThread
  • OscarMTMasterThread is now in the GlobalCache
    • The master std::thread is started in initializeGlobalCache(), and joined in globalEndRun()
    • Begin/end run is signalled to the master thread via enumeration variable and with mutex+condition_variable
    • Technically supports multiple runs in a job
      • Supported at the same level as in OscarProducer
      • E.g. geometry and magnetic field are obtained from EventSetup only during the first Run
  • All functionality in OscarProducer+RunManager should now be in OscarMTProducer+OscarMTMasterThread+RunManager+RunManagerMTWorker

I have tested with 100 MinBias events (wf 1.0) (in CMSSW_7_2_GEANT10_X_2014-07-01-0200+#4457+#4476)

  • OscarProducer and OscarMTProducer with 1 thread give identical results
    • OscarMTProducer with replay mode of RandomNumberGeneratorService checked manually with a couple of quantities
    • OscarMTProducer without replay mode checked with runEdmFileComparison.py
  • But OscarMTProducer with >= 2 threads gives different results for some events (randomly, roughly in 1 event in 10-20 events)
    • Something to be followed up

In addition I've tested with 200 TTBar events (wf 1302.0) that OscarMTProducer works technically at least up to 20 threads.

To ease testing with cmsDriver.py/runTheMatrix.py, I've included a customise-script to change all OscarProducers in a Process to OscarMTProducers.

cmsDriver.py ... --customise SimG4Core/Application/customiseMultithreadedSim.customiseMultithreadedSim --suffix "-n NUMTHREADS"
runTheMatrix.py ... --command="--customise SimG4Core/Application/customiseMultithreadedSim.customiseMultithreadedSim --suffix '-n NUMTHREADS'" --apply 0

Following the Geant4 threading model, there are quite a lot of new thread_local statics (especially in RunManagerMTWorker, but also elsewhere). At least the RunManagerMTWorker ones could probably be cleaned up with the "usual tricks" in the future.

@makortel
Copy link
Contributor Author

makortel commented Jul 3, 2014

Argh, just noticed that I forgot to squash a couple of commit, I'll reopen when done.

@makortel makortel closed this Jul 3, 2014
Suggested by @Dr15Jones, makes the stopG4()+join() time more
deterministic.
Feels dirty, but by far the easiest solution given everything was
static already.
…e() to take const SensitiveDetectorCatalog

Percolate the change through SensitiveDetector class hierarchy. Needed
to integrate Geant4MT to our framework.
The interface of G4UIsession has been changed slightly (or this has
never worked...), now ReceiveG4cout/cerr are properly overridden.
RunManagerMTInit::readES() has checks if geometry or magnetic field
change, and the checks make little sense if RunManagerMTInit's
lifetime is Run and not the whole program.
Helgrind reports got me read Geant4 code more carefully, and I noticed
that from worker threads one should set the geometry
WorkerDefineWorldVolume(). However, since we don't employ
G4(MT)RunManager, we have to call
G4TransportationManager::SetWorldForTracking() by ourselves.

WorkerDefineWorldVolume() seems to avoid writing to the (global) world
volume object, something that DefineWorldVolume() does.

Fixes some Helgrind warnings
Towards supporting multiple runs in a job.
Needed to support multiple runs in a job
… and stopThread()

Needed to support multiple runs in a job
…ng transitions from globalBeginRun/globalEndRun/globalEndJob
Mimicking the behaviour of RunManager::initG4().
@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 3, 2014

A new Pull Request was created by @makortel (Matti Kortelainen) for CMSSW_7_2_X.

Complete Geant4 MT integration

It involves the following packages:

Alignment/LaserAlignmentSimulation
SimG4CMS/Calo
SimG4CMS/CherenkovAnalysis
SimG4CMS/EcalTestBeam
SimG4CMS/FP420
SimG4CMS/Forward
SimG4CMS/HcalTestBeam
SimG4CMS/Muon
SimG4CMS/ShowerLibraryProducer
SimG4CMS/Tracker
SimG4Core/Application
SimG4Core/CustomPhysics
SimG4Core/GFlash
SimG4Core/Geometry
SimG4Core/MagneticField
SimG4Core/Notification
SimG4Core/Physics
SimG4Core/PhysicsLists
SimG4Core/SensitiveDetector

@civanch, @diguida, @mdhildreth, @cmsbuild, @nclopezo, @rcastello, @Degano can you please review it and eventually sign? Thanks.
@ghellwig, @namapane, @argiro this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 3, 2014

Pull request #4505 was updated. @civanch, @diguida, @mdhildreth, @cmsbuild, @nclopezo, @rcastello, @Degano can you please check and sign again.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 3, 2014

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 3, 2014

@civanch
Copy link
Contributor

civanch commented Jul 4, 2014

+1

@diguida
Copy link
Contributor

diguida commented Jul 4, 2014

+1

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 4, 2014

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_2_X IBs unless changes (tests are also fine).

ktf added a commit that referenced this pull request Jul 5, 2014
Complete Geant4 MT integration.
@ktf ktf merged commit 3758e94 into cms-sw:CMSSW_7_2_X Jul 5, 2014
@makortel makortel deleted the geant10mt_step2 branch October 20, 2016 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants