-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement sharding, other CPQ requirements #12
Open
smartin015
wants to merge
45
commits into
main
Choose a base branch
from
sharding
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ake test. Server still needs to implement basic commands
…qlite3 inmemory db, fix some behavior in load testing
…st seems to work OK, probably lossy
… python; fixed sync signing errors
…mand stability hopefully
…, logout link, device enrollment
…y, and use of request context for www handlers
… PSK generator, enable geo and timeline plotting plus testers, plumb through manual storage sync
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Continuous Print expects an ordered list of jobs. PP is currently designed with an unordered hashtable of jobs. We need to bridge the gap.
Stern-Brocot Tree based sorting
https://begriffs.com/posts/2018-03-20-user-defined-order.html#approach-3-true-fractions
Implement this method of selecting sortable factions in golang, and use it to assign an order to jobs that get inserted. Collisions must be handled gracefully (i.e. fallback to comparing job hash, also detect race conditions and somehow get the racing shards to agree).
Peers must listen on job addition requests to keep track of the upper/lower bound of the fractional sorting value, so that new jobs can be added appropriately.
Stats and job fetching
Jobs should have a set of tags that can be queried to get certain types of jobs. Additionally, there should be RPC handlers for getting the number of jobs between two rank values.
RPC based actions
Making a request and awaiting a response leads to simpler command behaviors, especially necessary when performing sequenced actions such as sharding off part of the job queue.
Need to create an
RPCBroker
interface which wraps pubsub and provides this blocking call behavior, with timeouts etc.Must also generalize the existing
poll
client behavior to allow multi-response requests over pubsub.