Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce model configs to abstract pairings of models and hardware #2194

Merged
merged 9 commits into from
Mar 24, 2023

Conversation

yk
Copy link
Collaborator

@yk yk commented Mar 24, 2023

  • introduces the concept of "model config" defined in model_configs.py that can be referenced by the frontend by name
  • adjusted dockerfiles to automatically determine model id and quantization from a given config name
  • introduced an OOM test to find out sequence length and batch size limits on different GPUs, results in notion

@github-actions
Copy link

pre-commit failed.
Please run pre-commit run --all-files locally and commit the changes.
Find more information in the repository's CONTRIBUTING.md

inference/worker/__main__.py Outdated Show resolved Hide resolved
inference/worker/__main__.py Outdated Show resolved Hide resolved
inference/worker/basic_hf_server.py Outdated Show resolved Hide resolved
inference/worker/get_model_config_prop.py Show resolved Hide resolved
inference/worker/interface.py Show resolved Hide resolved
inference/worker/run_worker_container.sh Show resolved Hide resolved
oasst-shared/oasst_shared/model_configs.py Show resolved Hide resolved
@yk yk enabled auto-merge (squash) March 24, 2023 21:26
@yk yk merged commit 0331e69 into main Mar 24, 2023
@yk yk deleted the model-ids-to-configs branch March 24, 2023 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants