clip: Fix llama-llava-clip-quantize-cli quantization error under CUDA backend #12566
+4
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t will cause ggml_fp16_to_fp32 to report an error when trying to access video memory. You need to switch to the CPU backend to run quantize. After the fix, it will automatically run in the CPU backend and will no longer be bound to CUDA.
ngxson
requested changes
Mar 25, 2025
examples/llava/clip.h
Outdated
| @@ -45,7 +45,7 @@ struct clip_context_params { | |||
| }; | |||
|
|
|||
| // deprecated, use clip_init | |||
| CLIP_API struct clip_ctx * clip_model_load(const char * fname, int verbosity); | |||
| CLIP_API struct clip_ctx * clip_model_load(const char * fname, const int verbosity=1, const bool use_gpu=true); | |||
There was a problem hiding this comment.
Do not change the signature of this function, many downstream binding will break.
Instead, use clip_init and set use_gpu via clip_context_params
There was a problem hiding this comment.
Also the const int verbosity=1 will fail to compile in C, some users use this header file in C projects.
…nd change the call in clip_model_quantize to clip_init.
ngxson
approved these changes
Mar 26, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Link to issue 12564. This PR provides a possible solution, which is to move all the quantization processes to the CPU backend. Since it is not a real inference process, the performance does not seem to be so sensitive.
Make sure to read the contributing guidelines before submitting a PR