Rebase of the CMSSW_7_1_THREADED_X branch on top of CMSSW_7_1_X #3461

…nously so must be atomic

This version uses a TBB task per Stream to process Events with a mutex used to protect the Source in a critical section. This is sufficient to test the thread safety of the framework internals.

The EventProcessor now uses the new C++11 std::exception_ptr, std::current_exception() and std:rethrow_exception() to capture a thrown exception in a thread processing an event and then transfer that exception to the main thread to be rethrown. We use an std::atomic to provide thread safe update of the std::exception_ptr member data in the case of multiple processing threads having an exception. In such a case, only one of the exceptions will be kept.

The SharedResourcesRegistry is the centralized place where resources are registered and where the framework can get SharedResourcesAcquirer instances. These two classes work together to allow safe sharing of non-thread safe resources between modules. A unit test is included to test the two classes.

… to interface In order to more easily accommodate unscheduled execution, I switched from using a mutex to a recursive_mutex. This will be less performant since the locks will be held longer than might be needed but it is easier to implement initially. A more performant way would be to release the locks temporarily while we are doing a ‘edm::Event::getBy*’ call.

Now uses a std::mutex to guarantee a module instance is only processing one event at a time. Also uses a edm::SharedResourcesAcquirer to get the legacy resource and therefore guarantee only one legacy module is running at a time.

…haredResource All ‘one’ style base classes now hold a SharedResourcesAcquirer and use a new virtual function ‘createAcquirer()’ which by default returns an empty SharedResourcesAcquirer instance. If the module declares the use of a ‘SharedResource’ then the ‘createAcquirer()’ method will be overridden and return an instance with the needed resources. To facility this, edm::one::imll::SharedResourcesUser had to be changed to a templated class.

The SharedResourcesUser class was just changed to a template. As with the other implementor types, this needs to be explicitly instantiated.

Use a global mutex to serialize access to all EventSetup modules and sources. This is overkill but guarantees that if any two modules/sources are sharing a non-thread safe resource we don’t have a problem. Changed two of the mutable to std::atomics since they are read/written outside of the critical section guarded by the mutex.

To allow a DelayedReader to synchronize with itself we want to use a SharedResourcesAcquirer. However, the normal procedure of declaring a resource and then only after all resources have been declared do we get an acquirer isn’t appropriate for this case. Therefore we provide a special resource just for this case.

If a DelayedReader needs to synchronize with itself (and possibly with the Source) the inheriting class can override the sharedResources_() method. The method is used when data products are requested from the DelayedReader. If the Source also needs to synchronize with the DelayedReader the inheriting class can override resourceSharedWithDelayedReader_() and return a SharedResourcesAcquirer which uses the same resource as the DelayedReader. The EventProcessor will acquire the resource before calling methods on the Source.

Given that we can’t be reading from the InputSource at the same time as we fill a branch via the RootDelayedReader, we implement the new APIs to allow both the PoolSource and the RootDelayedReader to share a ‘resource’.

Moved the sentry used to clear the event selector at the end of each event processing to be within the mutex. This makes sure all updates on that object are done in the same critical section.

…rom PoolSource

For ROOT 5 we must load all plugins before starting the threaded event processing. Therefore I added an option to turn on loading of all dictionaries up front. Also, ROOT requires us to call TThread::Initialize() in order to get ROOT to use internal mutexes.

…thod If you tell ROOT that it needs to be thread safe, it then expects all classes inheriting from TClassStreamer to implement the Generate() method. The method simply uses the copy constructor of the object to return a ‘new’ed version of the object.

We must follow ROOT’s rule which requires a dummy TThread instance be created in each operating system thread in order to create their own thread local storage. Given that we don’t know what thread a TBB task will be run, for right now we create a thread local static TThread on the function which is called from both the main thread and by TBB tasks in order to process events.

Instead of spreading around the different calls needed to setup ROOT for threading, all the methods are concentrated in InitRootHandlers. The EventProcessor then calls the correct virtual methods of the base class RootHandlers Service interface to handle the setup.

Disabled fast copying of ROOT files when we have multiple threads and multiple Streams. This must be done since fast copying requires the output file to record all the events from the input in the same order as stored in the input. This is not guaranteed when we do multiple threads.

The threaded framework requires Services to switch to a new collection of signals. To allow easier migration, the obsolete signals have been replaced with a class which causes an Exception to be thrown if their connect method is called. Had to change one unit test which was making use of the old signals.

Modifications to make the random number generator service work with the multithreaded Framework. The interface now requires a StreamID or LuminosityBlockIndex argument to the getEngine function. For each module, there will an engine for each stream and also each concurrent luminosity block instead of being just one per module. This is necessary to prevent data races. The service now uses the new callbacks for the threaded Framework. This required extensive revisions to the internals of the service and also to the related unit tests. If all modules use the new getEngine interface, the service will support multiple threads, forking (but not multiple threads and forking in the same job), and replay. Temporarily the old interface still works while clients are transitioned to the new interface. But modules using the old interface must be ONE or LEGACY type modules (not GLOBAL OR STREAM). If any modules use the old interface, then both replay and multiprocess forking will be broken.

…D branch

branch only. Affects unit tests only. Reference files needed updating plus minor cleanup. The change that caused this problem was cleanly merged from the 7_0_X branch, but I did not notice the unit test reference files needed to be different on the threaded branch. One conflicted and the other did not exist in 7_0_X.

With this change, modules that use the no argument form of the usesResource function will not run at the same time as any other module that has declared any shared resource or used the no argument usesResource function.

Replace a linear search for legacy resources by a data member that counts them. Also use the std::make_shared function instead of new.

The file contains one histogram used during reconstruction. Unfortunately, the file was written so long ago that ROOT's schema for the classes has changed. This is a problem for the threaded framework since ROOT 5 updates schema in a thread-unsafe manner when it opens a file and sees a new-to-it schema. By creating a new file containing the same information we can successfully preload the new class schemas (since they match the C++ code) at job initialization time. ROOT 6 is supposed to fix this problem.

The code which constructs ROOT Streamers is not thread safe. Unfortunately, Streamers normally get created the first time they are needed which would be too late for our case. This change forces all Streamers to be created, even if we do not need them for this job.

The global container of TStreamerInfos is not thread-safe. Therefore the ROOT team advised to create all TStreamerInfos needed for a job up front. We already create all TStreamerInfos for schemas corresponding to the present class layouts. However, a TFile can contain old schemas and will wants to create their TStreamerInfos the first time the class is read from the file. For our purposes, that is too late and therefore we need to force ROOT to create them at file open time.

std::out in its constructor.

Changed all tabs to be spaces to avoid problems between editors.

When requested to load a dictionary, we have to manipulate some Cint data structures to say what file it comes from. When used in a threaded environment, ROOT requires a lock be taken before accessing these structures.

ROOT will optimize Streamers such that adjoining member data of the same basic type are written/read together. Sometimes ROOT decides this optimization should not be applied and de-optimizes the Streamer. Unfortunately, the decision to de-optimize comes very late and can happen during the event loop and cause a data race. Therefore we tell ROOT, for now, to never optimize.

Fix problem when 7_1_X was merged into 7_1_THREADED_X. There was a conflict and some necessary changes were not merged. This is causing compilation to fail in 7_1_THREADED_X.

When we switched to ROOT 5.34.17 a new warning message appeared when TH1 is passed nbins <= 0. Since ROOT already sets the value to nbins=1 we can ignore this warning.

In the random number generator service, remove the getEngine function which takes no arguments. This function is replaced by new getEngine functions that take either a StreamID or a LuminosityBlockIndex as an argument. This comes after a long migration of all clients to the new interface. The new interace is designed to work with the multithreaded Framework. (Note that this also fixes a compilation error that resulted after a similar pull request was automatically merged from 7_1_X)

ROOT 5.34.17 has added additional warnings which we convert to exceptions. The warning about nbinsy <=0 is irrelevant since ROOT has a reasonable behavior in this case. Therefore, we want to ignore this warning.

The new thread-safety changes to ROOT make the need to try to force all Streamers into existence early unnecessary.

ROOT locks the same mutex before loading a shared library and in statics embedded in shared libaries. In the middle the operating system takes a lock to protect the shared library data structures. Therefore to avoid deadlock problems we must take the same ROOT lock before loading any shared libraries since we do not know apriori if the shared library being loaded links to a library which will take the lock at load time.

To avoid having the first lock order incorrect because the Cint lock is not yet created when PluginManager first calls R__LOCKGUARD, I switched to R__LOCKGUARD2 which will create that Cint lock if ROOT has decided that locks should be made but hasn't yet made them.

Forcing the creating of StreamerInfos at file open is no longer needed and it causes problems when we have done a 'drop on input' for a file and the items dropped can not be schema evolved. In that case the original code would cause an exception and it would kill the job. This change allows us to properly handle 'drop on input'.

The Geant4e based propagator is not presently thread safe. To avoid problems with the static analyzer we are removing it from the threaded branch. If the code is made thread safe, we will add it back.

The PolyFit3DParametrizedMagneticField is not presently thread safe. To avoid problems with the static analyzer we are removing it from the threaded branch. If the code is made thread safe, we will add it back.

The DQM service is not thread-safe and must be removed from configurations until it is changed.

The TkDetMap service is not thread-safe and violates CMSSW coding rules and must be removed from configurations until it is changed. Modules dependent upon TkDetMap have also been removed from configurations.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebase of the CMSSW_7_1_THREADED_X branch on top of CMSSW_7_1_X #3461

Rebase of the CMSSW_7_1_THREADED_X branch on top of CMSSW_7_1_X #3461

Commits on Apr 24, 2014