Refactor task scheduler #95
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Several improvements to the task scheduler: - More robust handling of large object allocation/deallocation (e.g. concact matrices are allocated and deallocated once in a thread-safe manner using std::call_once) - Make scheduler more general, so that tasks can refer to genomic intervals instead of entire chromosomes
robomics
force-pushed
the
refactor-scheduler
branch
from
baf375f
to
723172e
Compare
Rework contact sampling to deal with very small simulations (e.g. those involving a single LEF). With a single or few LEFs, it is possible (and even likely), that none of the LEFs is within bound when sampling interactions. In this case, the contact sampling loop would run indefinitely. Changes from this commit addresst this issue. IMPORTANT: changes from this commit changes MoDLE's output!
Instead of seeding PRNGs by hashing tasks, seed the PRNG once with the default/user-provided seed, then call the PRNG::jump(), which is equivalent to calling next() 2**64 times, and can be used to generate 2**64 non-overlapping sub-sequences of pseudo-random numbers.
Move most of the scheduling logic out of Simulation and into ContextManager. ContextManager now owns worker and IO threads, as well the task queues and exception ptrs. There are two task queues: 1. A queue for pending tasks 2. A queue for finished tasks The main thread pushes tasks onto queue #1. Worker threads pop tasks from queue #1 to then setup and run a simulation instance. Upon completion of a simulation instance, worker threads push the corresponding task onto queue #2. IO threads (for now only the thread responsible for writing interactions to a .cool file), consume tasks from queue #2 and perform IO operations when appropriate. The main thread must signal that no more tasks will be submitted to queue #1 by calling the shutdown() method. This method blocks and waits until all tasks have been processed and worker/IO threads have returned. The ContextManager is also responsible of keeping track of exceptions thrown inside worker/IO threads. ContextManager operator bool or exception_thrown() can be used to check whether any exception has been raised. All threads should regularly call one of these methods, and return immediately in case exceptions have been raised. The main thread should check if any exceptions have been raised by calling ContextManages::check_exceptions(), which internally calls shutdown() and rethrows the exceptions as appropriate.
robomics
force-pushed
the
refactor-scheduler
branch
from
d4db7bf
to
80f4057
Compare
Simulation was derived from Config to save some typing while iterating rapidly throughout different implementations in the early days. Those days are long gone, and so this hack is no longer required. Going forward, params from Config should be accessed by calling Simulation::c().
- Rename write_contacts_to_disk to simulate_io - Update simulate_io to write 1D LEF occupancy profiles as soon as all simulation instances for a given genomic interval are finished
robomics
force-pushed
the
refactor-scheduler
branch
2 times, most recently
from
f874948
to
d6250d7
Compare
robomics
force-pushed
the
refactor-scheduler
branch
from
d6250d7
to
6611e0b
Compare
robomics
force-pushed
the
refactor-scheduler
branch
from
98fc8bf
to
8409023
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Several improvements to the task scheduler:
This PR changes MoDLE's output, so the test datasets have also been updated.