-
Notifications
You must be signed in to change notification settings - Fork 189
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Device map feature for maestro models -qwen_2.5, florence_2 & paligemma_2 #179
base: develop
Are you sure you want to change the base?
Device map feature for maestro models -qwen_2.5, florence_2 & paligemma_2 #179
Conversation
Hi @AmazingK2k3 👋🏻 thank you so much for your PR. Could you please explain why you decided to drop the
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device="cpu"
)
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device="mps"
)
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device="cuda:0"
)
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device_map="auto"
)
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device_map={"": "cuda:0"}
) I think just |
Hey @SkalskiP, The main reason I dropped the device argument completely is that I felt having two arguments that dealt with handling devices
This will ultimately load the model across GPUs even if a specific device is requested, as stated in issue #176. I felt it would be much simpler to have a single argument dealing with the devices.
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device_map="cpu"
)
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device_map="mps"
)
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device_map="cuda:0"
)
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device_map="auto"
)
processor, model = load_model(
model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
device_map={"": "cuda:0"} # Not applicable to Florence 2
) Just one argument Let me know if this is okay or there is a better way to go about it |
Description
As discussed in this issue https://github.com/roboflow/maestro/issues/176, this PR implements the device map feature for loading all 3 models. No change in dependencies is required.
The 'device' hyperparameter was replaced by 'device map' to maintain consistency with huggingface and avoid confusion. It was also ensured in the Florence 2 model that the device map does not take in a dict input, eg: {"": "cuda:0"} and 'auto' directly assigns the device to an available device based on the already existing parse_device_spec() function.
For Qwen 2.5 and PaliGemma 2, the device map is directly passed to the loading of the models (from_pretrained), with the default set to 'auto'.
The docstring for the load_model() function for all 3 model checkpoints was updated to reflect the changes.
Type of change
Testing
Tested loading each model setting device map to different modes - 'auto', 'cuda', 'cpu'. In a cloud environment passing the cases.
I have read the CLA Document and I sign the CLA.