New feature requests

[XkunW]

New features we'd like to support:

• Alternative backend: SGLang
• Quantization

For any other features not mentioned here, please add them to this thread

[scarere]

Writing this here so that i don't forget. Track slurm job ids in a log file. Let users call vec-inf status list or something of the like to see all jobs started with vec-inf so that i don't need to copy paste the slurm-id or run squeue to find it.

[rohan-uiuc]

Are there plans to support downloading the model on the fly from HuggingFace, like vLLM supports? Would it be of interest if I open a Pull Request for it?

[XkunW]

Are there plans to support downloading the model on the fly from HuggingFace, like vLLM supports? Would it be of interest if I open a Pull Request for it?

We prefer our users not to download models to their own space to prevent duplications of the same model weights across different users, so that's not on our radar at the moment, but I can definitely see this to be useful for others. I'll add this on my todo list but if you would like to open a PR for it, feel free to do so:) We appreciate and welcome the contribution!

[rohan-uiuc]

Are there plans to support downloading the model on the fly from HuggingFace, like vLLM supports? Would it be of interest if I open a Pull Request for it?

We prefer our users not to download models to their own space to prevent duplications of the same model weights across different users, so that's not on our radar at the moment, but I can definitely see this to be useful for others. I'll add this on my todo list but if you would like to open a PR for it, feel free to do so:) We appreciate and welcome the contribution!

Sure, happy to. I had 2 approaches in mind, would you be able to share your preferred way of implementation?

1. Pass the model huggingface id in vLLM Arguments: Most minimal fix, will have to strip out that flag while creating the sbatch script, but is not a great DX because the user will have to pass in both the model name and then model id in args.
2. Maintain hugging face ids in the models.yaml file and fallback to it if model weights are not available locally. Better DX but bigger change and maintenance overhead.

Since I was testing, I already have a WIP PR with changes for approach 1. If approach 2 is preferred I can work on it and raise a PR. Please let me know.

[XkunW]

Closing this thread for cleanup, open individual issues for new requests going forward