[Feature] Support InstructBLIP #1685

techmonsterwang · 2023-07-02T12:51:13Z

Add InstructBLIP Implementation

codecov · 2023-07-02T13:11:27Z

Codecov Report

Patch coverage: 33.72% and project coverage change: -3.03 ⚠️

Comparison is base (f9dcae2) 68.16% compared to head (8c43282) 65.13%.

❗ Current head 8c43282 differs from pull request most recent head 807ce32. Consider uploading reports for the commit 807ce32 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #1685      +/-   ##
==========================================
- Coverage   68.16%   65.13%   -3.03%     
==========================================
  Files         295      332      +37     
  Lines       23372    25847    +2475     
  Branches     3713     4130     +417     
==========================================
+ Hits        15932    16836     +904     
- Misses       6880     8393    +1513     
- Partials      560      618      +58

Flag	Coverage Δ
unittests	`65.13% <33.72%> (-3.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmpretrain/apis/feature_extractor.py	`37.50% <0.00%> (ø)`
mmpretrain/apis/image_caption.py	`30.64% <0.00%> (ø)`
mmpretrain/apis/image_retrieval.py	`21.42% <0.00%> (ø)`
mmpretrain/apis/visual_grounding.py	`27.53% <0.00%> (ø)`
mmpretrain/apis/visual_question_answering.py	`25.67% <0.00%> (ø)`
mmpretrain/datasets/__init__.py	`60.46% <0.00%> (-13.83%)`	⬇️
mmpretrain/datasets/flickr30k_caption.py	`0.00% <0.00%> (ø)`
mmpretrain/datasets/flickr30k_retrieval.py	`0.00% <0.00%> (ø)`
mmpretrain/datasets/gqa_dataset.py	`0.00% <0.00%> (ø)`
mmpretrain/datasets/nocaps.py	`0.00% <0.00%> (ø)`
... and 65 more

... and 9 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

fangyixiao18 · 2023-07-13T07:39:47Z

configs/instructblip/README.md

@@ -0,0 +1,53 @@
+# MiniGPT4


Suggested change

# MiniGPT4

# InstructBLIP

fangyixiao18 · 2023-07-13T07:41:53Z

configs/instructblip/README.md

+
+## Models and results
+
+For Vicuna model, please refer to [MiniGPT-4 page](https://github.com/Vision-CAIR/MiniGPT-4) for preparation guidelines.


please change to instructblip page https://github.com/salesforce/LAVIS/tree/main/projects/instructblip

fangyixiao18 · 2023-07-13T07:43:46Z

configs/minigpt4/minigpt-4_vicuna-7b_caption.py

@@ -52,8 +52,13 @@
        'https://download.openmmlab.com/mmpretrain/v1.0/minigpt4/minigpt-4_qformer_20230615-1dfa889c.pth'  # noqa
    ),
    lang_encoder=dict(
-        type='AutoModelForCausalLM', name_or_path='YOUR_PATH_TO_VICUNA'),
-    tokenizer=dict(type='LlamaTokenizer', name_or_path='YOUR_PATH_TO_VICUNA'),
+        type='AutoModelForCausalLM',


plz restore the modification

fangyixiao18 · 2023-07-13T07:44:25Z

configs/instructblip/instructblip-vicuna7b_8xb32_caption.py

+    llm_tokenizer=dict(
+        type='LlamaTokenizer',
+        name_or_path=
+        '/mnt/petrelfs/share_data/liuyuan/llm_weights/vicuna_weights_7b'),


don't use our path

fangyixiao18 · 2023-07-13T07:44:39Z

configs/instructblip/instructblip-vicuna7b_8xb32_caption.py

+    text_backbone=dict(
+        type='AutoModelForCausalLM',
+        name_or_path=
+        '/mnt/petrelfs/share_data/liuyuan/llm_weights/vicuna_weights_7b'),


the same comment as above

fangyixiao18 · 2023-07-13T07:44:56Z

demo/Confusing-Pictures.jpg

why do you upload this image

fangyixiao18 · 2023-07-13T07:45:27Z

test.py

remove the this test file

fangyixiao18 · 2023-07-13T07:58:48Z

mmpretrain/models/multimodal/instructblip/instructblip_caption.py

+
+
+@MODELS.register_module()
+class InstructBlipCaption(BaseModel):


plz remove caption suffix and add 'task' arg in init

FDInSky · 2023-11-06T09:17:30Z

when will merge this? we need to use it. Thanks

techmonsterwang added 2 commits July 2, 2023 20:44

add instructblip

f680f4e

add instructblip

8a33a6a

techmonsterwang added 2 commits July 3, 2023 12:43

add instructblip

18d3a72

add instructblip

bb35d1f

techmonsterwang changed the base branch from main to dev July 5, 2023 07:52

fangyixiao18 changed the title ~~Instructblip~~ [Feature] Support InstructBLIP Jul 13, 2023

fangyixiao18 requested changes Jul 13, 2023

View reviewed changes

revert some modifications

807ce32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support InstructBLIP #1685

[Feature] Support InstructBLIP #1685

techmonsterwang commented Jul 2, 2023

codecov bot commented Jul 2, 2023 •

edited

fangyixiao18 Jul 13, 2023

fangyixiao18 Jul 13, 2023

fangyixiao18 Jul 13, 2023

fangyixiao18 Jul 13, 2023

fangyixiao18 Jul 13, 2023

fangyixiao18 Jul 13, 2023

fangyixiao18 Jul 13, 2023

fangyixiao18 Jul 13, 2023 •

edited

FDInSky commented Nov 6, 2023


		## Models and results

		For Vicuna model, please refer to [MiniGPT-4 page](https://github.com/Vision-CAIR/MiniGPT-4) for preparation guidelines.



		@MODELS.register_module()
		class InstructBlipCaption(BaseModel):

[Feature] Support InstructBLIP #1685

Are you sure you want to change the base?

[Feature] Support InstructBLIP #1685

Conversation

techmonsterwang commented Jul 2, 2023

codecov bot commented Jul 2, 2023 • edited

Codecov Report

fangyixiao18 Jul 13, 2023

Choose a reason for hiding this comment

fangyixiao18 Jul 13, 2023

Choose a reason for hiding this comment

fangyixiao18 Jul 13, 2023

Choose a reason for hiding this comment

fangyixiao18 Jul 13, 2023

Choose a reason for hiding this comment

fangyixiao18 Jul 13, 2023

Choose a reason for hiding this comment

fangyixiao18 Jul 13, 2023

Choose a reason for hiding this comment

fangyixiao18 Jul 13, 2023

Choose a reason for hiding this comment

fangyixiao18 Jul 13, 2023 • edited

Choose a reason for hiding this comment

FDInSky commented Nov 6, 2023

codecov bot commented Jul 2, 2023 •

edited

fangyixiao18 Jul 13, 2023 •

edited