Extend Phlex algorithm lifecycle to include pre- and post-execution phases "initialize" and "finalize".

[brettviren]

I'd like Phlex to consider to add support for calling two methods on an algorithm: initialize() and finalize() before and after the DFP graph execution (just after start and just before end of job) and from the main thread.

A templated compile-time check can be used so that algorithms that are simple functions or classes that lack one or both of these methods could be exempt from attempting to call their non-existent methods and without the need for inheritance.

This would help support Wire-Cell Toolkit and Geant4 in Phlex and probably other future integrations.

Are there any barriers to adding these two algorithm entry points?

[knoepfel]

Are there any barriers to adding these two algorithm entry points?

The barriers are conceptual: adding these entry points would mean that graph execution would need to be informed by not only data dependencies and the available concurrency, but it would also depend on algorithms that are used (presumably) only for establishing state...making some of the dependencies implicit. We are very hesitant to go this route until we can understand specifically what necessitates it. We have not yet seen a compelling case to add such hooks (although they have been considered).

Some options you might consider are below. Those options assume that the purpose of the "initialize" and "finalize" are to handle state used by the algorithm¹. Happy to follow up on anything.

Phlex support for functions bound to stateful objects

Functions bound to stateful objects can be registered with the framework. See here for an example of calling m.make<SomeClass>(ctor args...) where a framework-owned object of type SomeClass is created during registration. The algorithm registered by that code is the member function &even_fibonacci_numbers::only_even bound to the framework-owned object. That framework-owned object is then destroyed just before the program ends. It may be possible for you to get what you need by using SomeClass's constructor to serve as the "initialize", and SomeClass's destructor to serve as the "finalize".

This approach means no changes to Phlex proper, and it keeps stateful classes local to the Phlex module.

Recasting the "initialize" and "finalize" as data dependencies

Depending on what the initialize and finalize are for, it may be possible to introduce algorithms that prepare the state and pass that state explicitly as an in-memory-only data product to a downstream algorithm that requires it. This keeps the data dependencies explicit, thus making it easier to understand the flow of the program. How a "finalize" is implemented would depend on what it is for.

Resources

We have a reasonably mature design for Phlex resources. These can be helpful when needing to coordinate the execution of algorithms that depend on a shared resource that may not be thread-safe. Depending on what you need to do with the "initialize" and "finalize", these may be reasonable approaches.

Footnotes

1. In Phlex parlance, an algorithm is always a callable entity, generally a free function, a member function bound to an object, or a closure object generated from a lambda expression. ↩

[brettviren]

Thanks Kyle. I'll look more into m.make<>().

Just to be clear, I'm explicitly wanting these new methods to NOT be intertwined by TBB flow_graph execution.

0. Enter main()
1. call initialize() from the main thread on all that define it.
2. run flow_graph
3. flow_graph is truly and forever finished executing
4. call finalize() from the main thread on all that define it.
5. exit() from main().

I guess you also do not need to know about the flow graph, per se, just about objects that will go on to serve as node instances.

Here are some use cases:

• WCT "configurable" components are written with a guarantee that configure() is called serially in a single-threaded context. When wire-cell CLI and (I think) art provides main() this also means main-thread context. I do not yet know if any components need specifically main-thread or just single-thread. I currently kludge serial single-thread by calling WCT's configure phase the first time a Phlex algorithm bearing a WCT graph is executed. This requires a mutex as multiple WCT-bearing Phlex nodes can be executing concurrently.

• WCT "finalizable" components expect to have their terminate() called serially and prior to process tear down. It may actually be okay to soften this call the terminate phase from my WCT/Phlex node destructor (or the type given to m.make) but since existing WCT components always from with terminate() called prior to main() exit, some may be sensitive.

• Geant4 has thread-local variables and so must "live" entirely inside a single thread for its lifetime (ignoring its own internal MT'ing features). Integrating Geant4 in Phlex can't just use the first-call trick but needs to start Geant4 in its own (non-TBB pooled) thread. This also requires careful in-thread Geant4 tear down to avoid some main-thread G4 singleton destructors from destroying objects that were constructed in other-thread. This so far works for my G4 integration of edep-sim which uses G4RunManager but SHiP tried this trick and still has to leak G4 memory to avoid end-of-job crash (I suspect that may be due to SHiP using G4MTRunManager or my use just being lucky so far). With a guaranteed main-thread entry for construction and destruction of Geant4, I think these problems can be cleared up.

[olantwin]

From reading edep-sim, the run manager seems to be leaker there as well, so this seems to affect also the single-threaded G4 run manager, not just the MT one we use in SHiP.

Two points from my side:

• For me so far (ab)using constructors and destructors has been sufficient.
• The Geant4 threading model is hard to work with. Maybe some accommodations on the Phlex side can help, but I would like to also work with Geant4 to fix some of the pain points (and frequently frankly bugs) there (e.g. https://bugzilla-geant4.kek.jp/show_bug.cgi?id=2747, which they already fixed). Ideally, it would be good to have interfaces to use Geant4 without a run manager (or a custom one) or properly working TBB task manager with a shared pool.

[knoepfel]

We believe we have a solution for the Geant4 threading model using Phlex resources, which are still under development (see #730).