NeVa token fusion #9245

paul-gibbons · 2024-05-17T21:57:54Z

What does this PR do ?

Adding in token fusion via mlp_downsample adapter from VILA paper.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

…nsample height and weight Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

Slyne

LGTM.
No impact on my side.

nemo/collections/nlp/modules/common/text_generation_strategy.py

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

yaoyu-33 · 2024-05-31T02:17:25Z

nemo/collections/multimodal/data/neva/neva_dataset.py

@@ -265,6 +263,9 @@ def preprocess_multimodal(sources: dict, multimodal_cfg: dict, cur_token_len: in
    if media_type == 'video':
        num_patches *= multimodal_cfg['num_frames']

+    if multimodal_cfg['mm_mlp_adapter_type'] == 'mlp_downsample':
+        num_patches //= 4


did you handle padding somewhere else? for odd number dimension

Added in latest commit. I now check if media_tensor height or width divided by patch size is odd, if so then pad patch_dim +1. Same logic added to text_generation_strategy.

nemo/collections/multimodal/data/neva/neva_dataset.py

yaoyu-33 · 2024-05-31T02:19:40Z

nemo/collections/nlp/modules/common/text_generation_strategy.py

+                image_processor.crop_size['width'],
+            )
+
+        self.num_media_latents = (self.multimodal_cfg['crop_size'][0] // self.multimodal_cfg['patch_dim']) * (


consider odd number crop size

I've not seen odd number height or widths for crop_size in clip or siglip

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

into neva-token-fusion

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

* token fusion via mlp downsampling + media_type default fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * inference update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * adapter fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * config refactor, remove image_token_len dependency, transpose mlp_downsample height and weight Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * removing image_token_len in text generation strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fix patch_dim text generation Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * crop-size fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixing RGB reversal bug Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * crop_size default -> None in text_generation_strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * patch_dim padding for mlp_downsample Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * patch_dim padding update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * updating h/w patch_dim naming convention Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Signed-off-by: Boxiang Wang <boxiangw@nvidia.com>

* token fusion via mlp downsampling + media_type default fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * inference update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * adapter fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * config refactor, remove image_token_len dependency, transpose mlp_downsample height and weight Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * removing image_token_len in text generation strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fix patch_dim text generation Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * crop-size fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixing RGB reversal bug Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * crop_size default -> None in text_generation_strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * patch_dim padding for mlp_downsample Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * patch_dim padding update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * updating h/w patch_dim naming convention Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Signed-off-by: Jan Lasek <janek.lasek@gmail.com>

* token fusion via mlp downsampling + media_type default fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * inference update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * adapter fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * config refactor, remove image_token_len dependency, transpose mlp_downsample height and weight Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * removing image_token_len in text generation strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fix patch_dim text generation Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * crop-size fix Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * fixing RGB reversal bug Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * crop_size default -> None in text_generation_strategy Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * patch_dim padding for mlp_downsample Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * patch_dim padding update Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> * updating h/w patch_dim naming convention Signed-off-by: paul-gibbons <paul@gibbonspaul.com> * Apply isort and black reformatting Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> --------- Signed-off-by: paul-gibbons <paul@gibbonspaul.com> Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com> Co-authored-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

paul-gibbons added 3 commits May 10, 2024 14:03

token fusion via mlp downsampling + media_type default fix

2d910bd

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

inference update

ca06dfb

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

adapter fix

2800ad6

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

github-actions bot added NLP Multi Modal labels May 17, 2024

paul-gibbons and others added 8 commits May 28, 2024 18:14

Merge NeMo main branch into neva-token-fusion

53ca7a1

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

config refactor, remove image_token_len dependency, transpose mlp_dow…

76beb9b

…nsample height and weight Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Apply isort and black reformatting

438376f

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

removing image_token_len in text generation strategy

27a6ffe

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

fix patch_dim text generation

87aed0a

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

crop-size fix

a1e4af0

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

fixing RGB reversal bug

06b285b

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Apply isort and black reformatting

4f3acce

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

Slyne reviewed May 30, 2024

View reviewed changes

Merge branch 'main' into neva-token-fusion

666ab44

yaoyu-33 reviewed May 30, 2024

View reviewed changes

nemo/collections/nlp/modules/common/text_generation_strategy.py Outdated Show resolved Hide resolved

paul-gibbons and others added 2 commits May 30, 2024 18:11

crop_size default -> None in text_generation_strategy

b3d5414

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Apply isort and black reformatting

2cbbc8a

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

yaoyu-33 reviewed May 31, 2024

View reviewed changes

nemo/collections/multimodal/data/neva/neva_dataset.py Outdated Show resolved Hide resolved

yaoyu-33 reviewed May 31, 2024

View reviewed changes

paul-gibbons and others added 8 commits May 31, 2024 14:01

patch_dim padding for mlp_downsample

f0ef62d

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Merge branch 'neva-token-fusion' of https://github.com/paul-gibbons/NeMo

f6ba50b

into neva-token-fusion

Apply isort and black reformatting

ed4e9eb

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

patch_dim padding update

352f2bb

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

patch_dim padding update

3eb1201

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Apply isort and black reformatting

e052dcd

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

updating h/w patch_dim naming convention

fa0eeb2

Signed-off-by: paul-gibbons <paul@gibbonspaul.com>

Apply isort and black reformatting

b552e92

Signed-off-by: paul-gibbons <paul-gibbons@users.noreply.github.com>

yaoyu-33 added the Run CICD label Jun 3, 2024

yaoyu-33 approved these changes Jun 3, 2024

View reviewed changes

yaoyu-33 merged commit 48a2668 into NVIDIA:main Jun 3, 2024
130 checks passed

yaoyu-33 mentioned this pull request Jun 4, 2024

why we reverse the RGB channels in video processor #9346

Closed

ko3n1g mentioned this pull request Jul 18, 2024

Release 2.0.0rc1 #9786

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NeVa token fusion #9245

NeVa token fusion #9245

paul-gibbons commented May 17, 2024

Slyne left a comment

yaoyu-33 May 31, 2024

paul-gibbons May 31, 2024

yaoyu-33 May 31, 2024

paul-gibbons May 31, 2024

NeVa token fusion #9245

NeVa token fusion #9245

Conversation

paul-gibbons commented May 17, 2024

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Slyne left a comment

Choose a reason for hiding this comment

yaoyu-33 May 31, 2024

Choose a reason for hiding this comment

paul-gibbons May 31, 2024

Choose a reason for hiding this comment

yaoyu-33 May 31, 2024

Choose a reason for hiding this comment

paul-gibbons May 31, 2024

Choose a reason for hiding this comment