feat: add three new open clip roberta base models #860

OrangeSodahub · 2022-11-16T08:59:25Z

Goals : align with openclip v2.7.0
Changes :

Add three new models: roberta-ViT-B-32::laion2b-s12b-b32k xlm-roberta-base-ViT-B-32::laion5b-s13b-b90k, and xlm-roberta-large-ViT-H-14::frozen_laion5b_s13b_b90k;
Add LayernormFp32 (original Layernorm handles fp16) (Default precision: fp32 on cpu and fp16 on gpu);
Split original CLIP to TextTransformer, VisionTransformer and add _build_text_tower _build_vision_tower for seperately building;
Rearrange modules;
Fix bugs on flash attention (only use on cuda).
Docs will be updated in docs: add three new open clip roberta base models #862

* feat: bump openclip to v2.5.0 * fix: conflicts * fix: default fp32 on cpu and fp16 on gpu * feat: add two new models * fix: remove debug * fix: add roberta models (test) * fix: model name xlm * fix: (wip)

github-actions · 2022-11-16T09:00:19Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-16T09:30:30Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-16T09:35:10Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

codecov · 2022-11-16T09:50:26Z

Codecov Report

Merging #860 (04c130a) into main (e4717a3) will increase coverage by 0.10%.
The diff coverage is 91.74%.

@@            Coverage Diff             @@
##             main     #860      +/-   ##
==========================================
+ Coverage   80.28%   80.38%   +0.10%     
==========================================
  Files          22       22              
  Lines        1633     1448     -185     
==========================================
- Hits         1311     1164     -147     
+ Misses        322      284      -38

Flag	Coverage Δ
cas	`80.38% <91.74%> (+0.10%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/clip_server/model/flash_attention.py	`22.22% <ø> (+2.22%)`	⬆️
server/clip_server/model/pretrained_models.py	`98.41% <ø> (ø)`
server/clip_server/model/model.py	`75.09% <91.42%> (+4.98%)`	⬆️
client/clip_client/__init__.py	`100.00% <100.00%> (ø)`
server/clip_server/__init__.py	`100.00% <100.00%> (ø)`
server/clip_server/model/openclip_model.py	`93.10% <100.00%> (+3.44%)`	⬆️
server/clip_server/model/trt_utils.py	`56.04% <0.00%> (-27.48%)`	⬇️
server/clip_server/model/clip_trt.py	`69.38% <0.00%> (-16.33%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2022-11-16T10:54:20Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-16T11:10:25Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-16T11:12:20Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-16T15:09:54Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-17T06:00:15Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-17T06:07:31Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-20T16:21:29Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2022-11-21T07:39:09Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

numb3r3 · 2022-11-21T08:38:29Z

server/clip_server/model/flash_attention.py

+        assert not need_weights, "not allowed to return weights."
+        assert q.dtype in [
+            torch.float16,
+            torch.bfloat16,
+        ], f"flash attention only support torch.float16 or torch.bfloat16 but got {q.dtype}."
+        assert q.is_cuda, "flash attention only support cuda."


I would suggest removing these asserts. It's much safe, but degrading the performance a bit.

And what's more, from the function's parameter, seq_len. It seems that the flash-attention implementation can only be used for the text encoder. Is it can be applied to a vision transformer?

Yes it could. Every image tensor first convert to sentence-like tensor before fed into model.

server/clip_server/model/model.py

server/setup.py

numb3r3

LGTM

feat: bump openclip to v2.5.0 (#859)

5524f46

* feat: bump openclip to v2.5.0 * fix: conflicts * fix: default fp32 on cpu and fp16 on gpu * feat: add two new models * fix: remove debug * fix: add roberta models (test) * fix: model name xlm * fix: (wip)

github-actions bot added the size/xl label Nov 16, 2022

github-actions bot added the component/server label Nov 16, 2022

fix: remove roberta model

06ec06f

jina-ai deleted a comment from github-actions bot Nov 16, 2022

OrangeSodahub mentioned this pull request Nov 16, 2022

feat: bump openclip to v2.5.0 #859

Merged

fix: model name

7fcb813

fix: add :: to model name

c4beeca

github-actions bot added size/l and removed size/xl labels Nov 16, 2022

ZiniuYu force-pushed the bump-openclip-v2.50 branch from 1ebe483 to c4beeca Compare November 16, 2022 10:53

github-actions bot added size/xl and removed size/l labels Nov 16, 2022

fix: add transformers

5fbfb57

fix: remove is_init_value

d5dd1ce

fix: remove transformers

d937f13

github-actions bot added area/cicd area/housekeeping labels Nov 16, 2022

fix: not use flash-attn on cpu

fe2745b

fix: add assert description

7ddd51b

fix: typo

5478afc

fix: refactor

ba3aa44

github-actions bot added size/l and removed size/xl labels Nov 21, 2022

fix: refactor

2b7af82

github-actions bot added size/xl and removed size/l labels Nov 21, 2022

fix: refactor

06ad96a

github-actions bot added size/l and removed size/xl labels Nov 21, 2022

fix: type hint

8ef2090

ZiniuYu marked this pull request as ready for review November 21, 2022 08:02

numb3r3 requested changes Nov 21, 2022

View reviewed changes

ZiniuYu and others added 2 commits November 21, 2022 18:57

fix: address comments

8a36b8a

fix: visiontransformer

cf9595d

OrangeSodahub force-pushed the bump-openclip-v2.50 branch from 460622f to cf9595d Compare November 21, 2022 12:24

OrangeSodahub added 2 commits November 21, 2022 20:35

fix: d_model and n_head

e43990b

fix: texttransformer

1bf83ad

OrangeSodahub force-pushed the bump-openclip-v2.50 branch from 88fd2cf to 1bf83ad Compare November 21, 2022 13:31

ZiniuYu added 3 commits November 21, 2022 22:25

fix: change model to fix oom gh action

f918d65

fix: class init and param name

d910ee8

fix: black

423f788

ZiniuYu changed the title ~~feat: bump openclip to v2.5.0~~ feat: add three new clip roberta base models Nov 21, 2022

ZiniuYu changed the title ~~feat: add three new clip roberta base models~~ feat: add three new open clip roberta base models Nov 21, 2022

chore: bump open-clip version

04c130a

numb3r3 approved these changes Nov 29, 2022

View reviewed changes

numb3r3 merged commit f251539 into main Nov 29, 2022

numb3r3 deleted the bump-openclip-v2.50 branch November 29, 2022 06:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add three new open clip roberta base models #860

feat: add three new open clip roberta base models #860

OrangeSodahub commented Nov 16, 2022 •

edited by ZiniuYu

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

codecov bot commented Nov 16, 2022 •

edited

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 17, 2022

github-actions bot commented Nov 17, 2022

github-actions bot commented Nov 20, 2022

github-actions bot commented Nov 21, 2022

numb3r3 Nov 21, 2022

numb3r3 Nov 21, 2022

OrangeSodahub Nov 21, 2022

numb3r3 left a comment

feat: add three new open clip roberta base models #860

feat: add three new open clip roberta base models #860

Conversation

OrangeSodahub commented Nov 16, 2022 • edited by ZiniuYu

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

codecov bot commented Nov 16, 2022 • edited

Codecov Report

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 16, 2022

github-actions bot commented Nov 17, 2022

github-actions bot commented Nov 17, 2022

github-actions bot commented Nov 20, 2022

github-actions bot commented Nov 21, 2022

numb3r3 Nov 21, 2022

Choose a reason for hiding this comment

numb3r3 Nov 21, 2022

Choose a reason for hiding this comment

OrangeSodahub Nov 21, 2022

Choose a reason for hiding this comment

numb3r3 left a comment

Choose a reason for hiding this comment

OrangeSodahub commented Nov 16, 2022 •

edited by ZiniuYu

codecov bot commented Nov 16, 2022 •

edited