Skip to content

Add PaliGemma 2 mix checkpoints #2106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 20, 2025
Merged

Add PaliGemma 2 mix checkpoints #2106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a8e8066
Update pali_gemma_presets.py
bebechien Feb 20, 2025
d693b7a
Update pali_gemma_presets.py
bebechien Feb 20, 2025
0d21001
Update convert_pali_gemma2_checkpoints.py
bebechien Feb 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 93 additions & 3 deletions keras_hub/src/models/pali_gemma/pali_gemma_presets.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,96 @@
},
"kaggle_handle": "kaggle://keras/paligemma2/keras/pali_gemma2_ft_docci_10b_448/2",
},
"pali_gemma2_mix_3b_224": {
"metadata": {
"description": (
"3 billion parameter, image size 224, 27-layer for "
"SigLIP-So400m vision encoder and 26-layer Gemma2 2B lanuage "
"model. This model has been fine-tuned on a wide range of "
"vision-language tasks and domains."
),
"params": 3032094960,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
},
"kaggle_handle": "kaggle://keras/paligemma2/keras/pali_gemma2_mix_3b_224/2",
},
"pali_gemma2_mix_3b_448": {
"metadata": {
"description": (
"3 billion parameter, image size 448, 27-layer for "
"SigLIP-So400m vision encoder and 26-layer Gemma2 2B lanuage "
"model. This model has been fine-tuned on a wide range of "
"vision-language tasks and domains."
),
"params": 3032979696,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
},
"kaggle_handle": "kaggle://keras/paligemma2/keras/pali_gemma2_mix_3b_448/2",
},
"pali_gemma2_mix_10b_224": {
"metadata": {
"description": (
"10 billion parameter, image size 224, 27-layer for "
"SigLIP-So400m vision encoder and 42-layer Gemma2 9B lanuage "
"model. This model has been fine-tuned on a wide range of "
"vision-language tasks and domains."
),
"params": 9662409456,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
},
"kaggle_handle": "kaggle://keras/paligemma2/keras/pali_gemma2_mix_10b_224/2",
},
"pali_gemma2_mix_10b_448": {
"metadata": {
"description": (
"10 billion parameter, image size 448, 27-layer for "
"SigLIP-So400m vision encoder and 42-layer Gemma2 9B lanuage "
"model. This model has been fine-tuned on a wide range of "
"vision-language tasks and domains."
),
"params": 9663294192,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
},
"kaggle_handle": "kaggle://keras/paligemma2/keras/pali_gemma2_mix_10b_448/2",
},
"pali_gemma2_mix_28b_224": {
"metadata": {
"description": (
"28 billion parameter, image size 224, 27-layer for "
"SigLIP-So400m vision encoder and 46-layer Gemma2 27B lanuage "
"model. This model has been fine-tuned on a wide range of "
"vision-language tasks and domains."
),
"params": 27650192112,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
},
"kaggle_handle": "kaggle://keras/paligemma2/keras/pali_gemma2_28b_mix_224/2",
},
"pali_gemma2_mix_28b_448": {
"metadata": {
"description": (
"28 billion parameter, image size 448, 27-layer for "
"SigLIP-So400m vision encoder and 46-layer Gemma2 27B lanuage "
"model. This model has been fine-tuned on a wide range of "
"vision-language tasks and domains."
),
"params": 27650192112,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
},
"kaggle_handle": "kaggle://keras/paligemma2/keras/pali_gemma2_28b_mix_448/2",
},
"pali_gemma2_pt_3b_224": {
"metadata": {
"description": (
Expand Down Expand Up @@ -181,7 +271,7 @@
"model. This model has been pre-trained on a mixture of "
"datasets."
),
"params": 9662409456,
"params": 27650192112,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
Expand All @@ -196,7 +286,7 @@
"model. This model has been pre-trained on a mixture of "
"datasets."
),
"params": 9663294192,
"params": 27650192112,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
Expand All @@ -211,7 +301,7 @@
"model. This model has been pre-trained on a mixture of "
"datasets."
),
"params": 9666833136,
"params": 27650192112,
"official_name": "PaliGemma2",
"path": "pali_gemma2",
"model_card": "https://www.kaggle.com/models/google/paligemma-2",
Expand Down
6 changes: 6 additions & 0 deletions tools/checkpoint_conversion/convert_pali_gemma2_checkpoints.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,12 @@
"pali_gemma2_10b_ft_docci_448": (
"google/paligemma-2/jax/paligemma2-10b-ft-docci-448"
),
"pali_gemma2_3b_mix_224": "google/paligemma-2/jax/paligemma2-3b-mix-224",
"pali_gemma2_3b_mix_448": "google/paligemma-2/jax/paligemma2-3b-mix-448",
"pali_gemma2_10b_mix_224": "google/paligemma-2/jax/paligemma2-10b-mix-224",
"pali_gemma2_10b_mix_448": "google/paligemma-2/jax/paligemma2-10b-mix-448",
"pali_gemma2_28b_mix_224": "google/paligemma-2/jax/paligemma2-28b-mix-224",
"pali_gemma2_28b_mix_448": "google/paligemma-2/jax/paligemma2-28b-mix-448",
"pali_gemma2_3b_pt_224": "google/paligemma-2/jax/paligemma2-3b-pt-224",
"pali_gemma2_3b_pt_448": "google/paligemma-2/jax/paligemma2-3b-pt-448",
"pali_gemma2_3b_pt_896": "google/paligemma-2/jax/paligemma2-3b-pt-896",
Expand Down
Loading