Skip to content

Conversation

james77777778
Copy link
Collaborator

@james77777778 james77777778 commented Oct 30, 2024

This PR:

  • Add tests for CLIPBackbone
  • Add conversion script for CLIP
  • Introduces 8 presets for CLIP from OpenAI and OpenCLIP repo
  • Fixes several dtype bugs in CLIP
  • Adds support for sublayer and submodel in init_kwargs in run_backbone_test

@divyashreepathihalli

Some results (from conversion script):

Preset Modeling difference
clip_vit_base_patch16 0.009210587
clip_vit_base_patch32 0.0061130524
clip_vit_large_patch14 0.008178234
clip_vit_large_patch14_336 0.004854679
clip_vit_b_32_laion2b_s34b_b79k 0.02490139
clip_vit_h_14_laion2b_s32b_b79k 0.0014100075
clip_vit_g_14_laion2b_s12b_b42k 1.2397766e-05
clip_vit_bigg_14_laion2b_39b_b160k 1.04904175e-05

Preprocessing difference: 0.095298484

Copy link
Collaborator

@divyashreepathihalli divyashreepathihalli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@divyashreepathihalli divyashreepathihalli added the kokoro:force-run Runs Tests on GPU label Oct 30, 2024
@kokoro-team kokoro-team removed the kokoro:force-run Runs Tests on GPU label Oct 30, 2024
@james77777778
Copy link
Collaborator Author

james77777778 commented Oct 31, 2024

I forgot to register CLIP presets which caused the CI to fail. It should be fixed now.

EDITED:
The GPU CI failure should be irrelevant

@james77777778 james77777778 added the kokoro:force-run Runs Tests on GPU label Oct 31, 2024
@kokoro-team kokoro-team removed the kokoro:force-run Runs Tests on GPU label Oct 31, 2024
@james77777778 james77777778 added the kokoro:force-run Runs Tests on GPU label Oct 31, 2024
@kokoro-team kokoro-team removed the kokoro:force-run Runs Tests on GPU label Oct 31, 2024
@divyashreepathihalli divyashreepathihalli added the kokoro:force-run Runs Tests on GPU label Oct 31, 2024
@kokoro-team kokoro-team removed the kokoro:force-run Runs Tests on GPU label Oct 31, 2024
@divyashreepathihalli divyashreepathihalli merged commit edda678 into keras-team:master Nov 1, 2024
10 checks passed
@james77777778 james77777778 deleted the add-clip-presets branch November 3, 2024 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants