-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using Source = Local for Base Model #347
Comments
Hey @silveranalytics, thanks for raising this issue. We definitely want to make it easier for folks to use LoRAX without needing to go to HF. Happy to take some time to investigate this. |
@silveranalytics I ran into a similar error. I added a "local" condition to elif source == "hub":
config_dict, _ = PretrainedConfig.get_config_dict(
model_id, revision=revision, trust_remote_code=trust_remote_code
)
elif source == "local":
config_dict, _ = PretrainedConfig.get_config_dict(
model_id, revision=revision, trust_remote_code=trust_remote_code
)
else: After the change I was able to specify |
This is a must have for us. We need the ability to self-host. We cannot go out to the internet to get the base model weights. We need to load them from disk |
Having an issue with local models as well :( |
As I understand, this fix is still in work now. Could you advise when it will be ready? |
We were able to get the base model loaded from a local source using
|
Feature request
I only see source=local available for the adapters, is this the case?
Even with the models cached/pointing to it locally, there is still a callout to HF without changing the source from 'hub'.
Motivation
My ultimate goal is to run offline.
docker run -e RUST_BACKTRACE=full --gpus '"device=3"' --network none --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/predibase/lorax:latest --model-id /data/models--mistralai--Mistral-7B-Instruct-v0.1/
Your contribution
I've tried to add the option in get_model but I am a novice.
Even with adding the "local", I still get an error:
The text was updated successfully, but these errors were encountered: