Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement sharding, other CPQ requirements #12

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open

Conversation

smartin015
Copy link
Owner

@smartin015 smartin015 commented Dec 22, 2022

Continuous Print expects an ordered list of jobs. PP is currently designed with an unordered hashtable of jobs. We need to bridge the gap.

Stern-Brocot Tree based sorting

https://begriffs.com/posts/2018-03-20-user-defined-order.html#approach-3-true-fractions

Implement this method of selecting sortable factions in golang, and use it to assign an order to jobs that get inserted. Collisions must be handled gracefully (i.e. fallback to comparing job hash, also detect race conditions and somehow get the racing shards to agree).

Peers must listen on job addition requests to keep track of the upper/lower bound of the fractional sorting value, so that new jobs can be added appropriately.

Stats and job fetching

Jobs should have a set of tags that can be queried to get certain types of jobs. Additionally, there should be RPC handlers for getting the number of jobs between two rank values.

RPC based actions

Making a request and awaiting a response leads to simpler command behaviors, especially necessary when performing sequenced actions such as sharding off part of the job queue.

Need to create an RPCBroker interface which wraps pubsub and provides this blocking call behavior, with timeouts etc.

Must also generalize the existing poll client behavior to allow multi-response requests over pubsub.

…ake test. Server still needs to implement basic commands
…qlite3 inmemory db, fix some behavior in load testing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant