Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime updates #237

Draft
wants to merge 43 commits into
base: main
Choose a base branch
from
Draft

Runtime updates #237

wants to merge 43 commits into from

Conversation

edyounis
Copy link
Member

@edyounis edyounis commented Mar 28, 2024

Improves the runtime by addressing some shortcomings that cause issues in rare situations and implementing features that provide quality of life improvements:

  • Splits the single-threaded worker into two threads: one thread does work, while one consumes incoming messages. This prevents the rx buffer from filling up and blocking other parts of the system while the worker is working on a long job.
  • Added read receipts to the waiting messages to create a "happens-before" relationship with certain messages. This ensures eventual consistency with idle worker tracking through the system and avoids a previously allowed race condition. Moreover, the manager can now always adjust their idle count to an accurate amount.
  • Workers now send messages instantly rather than waiting until their current job has finished or enters an awaited state. This allows for child tasks to start quicker and log messages to appear instantly.
  • Improved Serialization
    • Circuit serialization now uses dill internally (even if called with pickle) for gates that cannot be pickled normally
    • Circuit serialization uses pickle (even if called with dill) to pickle gates that can be pickled normally
    • Workflow serialization now uses dill internally, allowing lambdas to be passed as pass arguments (and more)
    • Additionally, workflows using dill now allows passes defined in __main__ to be used without exporting to another file
    • RuntimeTasks only deserialize fnargs at destination now rather than at every node in the runtime
    • RuntimeTasks use dill to serialize fnargs allowing lambdas to be passed to children tasks (and more)
  • Compilers export their sys path to the runtime, allowing normally unreachable files to be found, depending on file system similarities
  • Fixes: Harmless Race Condition in AttachedServer Cleanup #181
  • Fixes: Attached servers and workers importing numpy before setting BLAS threads to 1
  • Fixes: Workflow copy constructor not setting all necessary variables
  • Address: Print warning during attached server connection issues #211

TODO:

bqskit/runtime/__init__.py Outdated Show resolved Hide resolved
bqskit/runtime/base.py Outdated Show resolved Hide resolved
bqskit/runtime/base.py Outdated Show resolved Hide resolved
bqskit/runtime/base.py Show resolved Hide resolved
bqskit/runtime/base.py Outdated Show resolved Hide resolved
bqskit/runtime/worker.py Outdated Show resolved Hide resolved
bqskit/runtime/worker.py Outdated Show resolved Hide resolved
bqskit/runtime/worker.py Outdated Show resolved Hide resolved
bqskit/runtime/worker.py Outdated Show resolved Hide resolved
bqskit/runtime/worker.py Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants