Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DESIGN] auction-like priorities for servers #8

Closed
justheuristic opened this issue Jun 12, 2022 · 3 comments
Closed

[DESIGN] auction-like priorities for servers #8

justheuristic opened this issue Jun 12, 2022 · 3 comments
Assignees

Comments

@justheuristic
Copy link
Collaborator

[for the record: this was proposed by @TimDettmers ]

Currently, hivemind-server treats all requests on a first come first served basis.
If we want to reward active participants with faster inference/training, we could change that into an auction.

Here's how client-server interaction looks like:

  • server gives client its stats, the current-highest bid, and maybe some metadata for bidding, e.g. the lowest serviced bids over last T seconds
  • client makes a bid - and signs it in such a way that it becomes a commitment ( see [RESEARCH] LM API merit system #6 )
  • in TaskPool.iterate_minibatches, server will now generate minibatches in the order of highest bid first
  • in TaskPool.priority, server will now set pool's priority based on highest bid in the pool, instead of wait time

As suggested by @GreenFatGuy , we need to think through how to deal with situations when low bids on high-demand servers won't ever be processed, and will hence take up memory on both client and server. First order solution: add absolute expiration time to each request, drop requests that hit expiration time.

@justheuristic
Copy link
Collaborator Author

justheuristic commented Jun 19, 2022

currently, @GreenFatGuy investigates how best to implement that

@justheuristic justheuristic mentioned this issue Jun 20, 2022
32 tasks
@justheuristic
Copy link
Collaborator Author

Server-side parts implemented in #47

@justheuristic
Copy link
Collaborator Author

[next step: let users actually designate these priorities, to be added as a separate issue]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants