Use ordered data structure to keep track of queue processes #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* A dictionary is unordered, this could cause issues when joining processes since this operation needs to be done in order * Using a tuple is the best option here, since it is immutable and the structure is simple enough
Wason1797
deleted the
bugfix/use-ordered-data-structure-to-leep-track-of-queue-processes
branch
Wason1797
added a commit
that referenced
this pull request
Jan 3, 2022
* Use ordered data structure to keep track of queue processes (#7) * Fix potential sync bugs with processes being joined out of order * A dictionary is unordered, this could cause issues when joining processes since this operation needs to be done in order * Using a tuple is the best option here, since it is immutable and the structure is simple enough * push rolling releases with rc branch, not develop * bump package version to 0.0.2rc
Wason1797
added a commit
that referenced
this pull request
Jan 3, 2022
* Create 0.0.2rc (#8) * Use ordered data structure to keep track of queue processes (#7) * Fix potential sync bugs with processes being joined out of order * A dictionary is unordered, this could cause issues when joining processes since this operation needs to be done in order * Using a tuple is the best option here, since it is immutable and the structure is simple enough * push rolling releases with rc branch, not develop * bump package version to 0.0.2rc * bump package version to 0.0.2
Wason1797
added a commit
that referenced
this pull request
Oct 23, 2022
* Use ordered data structure to keep track of queue processes (#7) * Fix potential sync bugs with processes being joined out of order * A dictionary is unordered, this could cause issues when joining processes since this operation needs to be done in order * Using a tuple is the best option here, since it is immutable and the structure is simple enough * push rolling releases with rc branch, not develop * Create 0.0.2rc (#8) * Use ordered data structure to keep track of queue processes (#7) * Fix potential sync bugs with processes being joined out of order * A dictionary is unordered, this could cause issues when joining processes since this operation needs to be done in order * Using a tuple is the best option here, since it is immutable and the structure is simple enough * push rolling releases with rc branch, not develop * bump package version to 0.0.2rc * add mypy configuration file * I decided to start using mypy to typecheck the library * supress flake8 errors in init * improve inner library organization * Move the contents of main.py to automator.py so the naming was clearer * Also move the example outside the src folder of the library as it is not library code * Add support to insert data to a queue other than the initial one * Fix bug in the validate_non_empty_args helper * add a reset() api to clear the QueueAutomator state * Create MultiprocessMaybe * This multiprocessing helper uses the Maybe pattern as inspiration to chain several function calls using multiprocessing * It wraps a QueueAutomator instance and provides a different style API which could be cleaner than having a bunch of decorators and manually defining queue names. * This is a work in progress, and it is improving with every iteration * Add example for MultiprocessingMaybe * Showcase the MultiprocessingMaybe capabilities in an example -> insert to intermediate queues -> replace None or values matching a predicate with the default value -> Prevent crashes for None values not executing the function when the input matches a predicate * Add tests for QueueAutomator * Include the first round of tests for the queue automator, this will ensure that the main functionality does not break if changed * Update mypy configurations * This is to exclude build and dist directories from typechecking * Add docstrings to MultiprocessMaybe * Update rc version * Update Readme * Include Multiprocessing maybe examples * Update package version to publish
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix potential sync bugs with processes being joined out of order
A dictionary is unordered, this could cause issues when joining processes since this operation needs to be done in order
Using a tuple is the best option here, since it is immutable and the structure is simple enough
Push rolling releases with rc branch, not develop