New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Planned/Potential of significant work #1819

Open

9 tasks

Narsil opened this issue Apr 26, 2024 · 0 comments

Collaborator

Narsil commented Apr 26, 2024

Fp8 kv-cache
Kv-cache prefix reuse
Grammar constrained speedup
torch.compile like speedups
Simple one-liner pip install
Multi lora support (lorax kind of)
Marlin quantization
Exl2 quants (non gptq 2, 3, 4.5 bpw)
Adding more documentation/help regarding getting production grafana dashboards easily setup.

The text was updated successfully, but these errors were encountered:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment