Skip to content

fix clip allocation size error for 81-series vlm models #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Dec 3, 2024
Merged

fix clip allocation size error for 81-series vlm models #33

Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d6c0627
Merge pull request #7 from NexaAI/master
zhiyuan8 Nov 5, 2024
b535cd9
Merge pull request #9 from NexaAI/master
zhiyuan8 Nov 5, 2024
5f2d958
Merge pull request #19 from NexaAI/master
zhiyuan8 Nov 11, 2024
89bcf5a
Merge pull request #21 from NexaAI/master
zhiyuan8 Nov 12, 2024
bb33473
Merge pull request #23 from NexaAI/david/vulkan2
zhiyuan8 Nov 12, 2024
fe792d6
Merge pull request #26 from NexaAI/master
zhiyuan8 Nov 15, 2024
fd2c582
remove reference interface from extern C in qwen2audio examples
Nov 21, 2024
7589158
expose omni_context_params struct
Nov 21, 2024
71b563e
Merge branch 'weili/dev' of github.com:NexaAI/llama.cpp into weili/dev
Dec 3, 2024
be54cb0
bug fix
Dec 3, 2024
ca7e8ef
fix clip_n_patch() allocation size error for 81-series omni-vlm models
Dec 3, 2024
07c7ff3
Merge branch 'weili/dev' of github.com:NexaAI/llama.cpp into weili/dev
Dec 3, 2024
b86cded
remove iostream header
Dec 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 14 additions & 10 deletions examples/omni-vlm/clip.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1927,16 +1927,20 @@ int clip_n_patches(const struct clip_ctx * ctx) {

int n_patches = (params.image_size / params.patch_size) * (params.image_size / params.patch_size);

if (ctx->proj_type == PROJECTOR_TYPE_LDP || ctx->proj_type == PROJECTOR_TYPE_LDPV2) {
n_patches /= 4;
} else if (ctx->proj_type == PROJECTOR_TYPE_RESAMPLER) {
if (ctx->minicpmv_version == 2) {
n_patches = 96;
}
else if (ctx->minicpmv_version == 3) {
n_patches = 64;
}
}
if(ctx->omni_vlm_ver_type == omni_vlm_version_type::VLM_81_OCR
|| ctx->omni_vlm_ver_type == omni_vlm_version_type::VLM_81_INSTRUCT) {
n_patches /= 9;
}
// if (ctx->proj_type == PROJECTOR_TYPE_LDP || ctx->proj_type == PROJECTOR_TYPE_LDPV2) {
// n_patches /= 4;
// } else if (ctx->proj_type == PROJECTOR_TYPE_RESAMPLER) {
// if (ctx->minicpmv_version == 2) {
// n_patches = 96;
// }
// else if (ctx->minicpmv_version == 3) {
// n_patches = 64;
// }
// }

return n_patches;
}
Expand Down