Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix routing through relay, default network RPS, --token, logging, readme #399

Merged
merged 10 commits into from
Jul 22, 2023

Conversation

borzunov
Copy link
Collaborator

@borzunov borzunov commented Jul 21, 2023

  • Hide GeneratorExit in _iterate_inference_steps()
  • Update README.md about --public_name
  • Use .from_pretrained(..., use_auth_token=token) instead of token=token
    until it's fully supported across HF libs
  • Use default network speed 25 Mbit/s
  • Apply relay penalty in max-throughput routing
  • Replace RPS with "tokens/sec per block" in logs
  • Increase default expiration

@borzunov borzunov changed the title Hide GeneratorExit in _iterate_inference_steps() Improve server logging and readme Jul 22, 2023
@borzunov borzunov changed the title Improve server logging and readme Fix --token and server logging, improve readme Jul 22, 2023
@borzunov borzunov changed the title Fix --token and server logging, improve readme Fix routing through relay, default network RPS, --token, logging, readme Jul 22, 2023
@borzunov borzunov merged commit 8666653 into main Jul 22, 2023
7 checks passed
@borzunov borzunov deleted the minor-things branch July 22, 2023 14:27
borzunov added a commit that referenced this pull request Jul 22, 2023
The value is chosen as some safe value below average at https://health.petals.dev/

Note that if a server uses relays, the effective throughput will be further divided by 2 (see #399).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant