New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat] Add ViT modules #1099

Closed

Ryan-Qiyu-Jiang wants to merge 20 commits into gh/ryan-qiyu-jiang/4/base from gh/ryan-qiyu-jiang/4/head

Contributor

Ryan-Qiyu-Jiang commented Sep 17, 2021 •

edited

Stack from ghstack:

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: D31117429


          [feat] Add ViT modules

90c8657

Add modified huggingface ViT modules from megaVLT
supporting attention masks

[ghstack-poisoned]

facebook-github-bot added the CLA Signed label

This was referenced Sep 17, 2021

[feat] Add ViLT transformer heads #1096

Closed

[feat] Add ViLT image text processors #1097

Closed

[chore] Bump transformers version to 4.5.1 #1098

Closed

[feat] Add ViLT encoder #1100

Closed

[feat] Add ViLT image and text embeddings #1101

Closed

[feat] Add ViLT model #1102

Closed

[feat] Add ViLT yaml configs #1103

Closed


          Update on "[feat] Add ViT modules"

c053950

Add modified huggingface ViT modules from megaVLT
supporting attention masks

[ghstack-poisoned]

Ryan-Qiyu-Jiang mentioned this pull request

[feat] Add pretraining support to ViLT #1106

Closed

Ryan-Qiyu-Jiang added a commit to Ryan-Qiyu-Jiang/mmf that referenced this pull request


          [feat] Add ViT modules

d53181b

Add modified huggingface ViT modules from megaVLT
supporting attention masks

ghstack-source-id: dbb94a245b20194b1b906973124a66171c32b12e
Pull Request resolved: facebookresearch#1099

Contributor Author

Ryan-Qiyu-Jiang commented Sep 22, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.


          Update on "[feat] Add ViT modules"

50a2fdb

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Ryan-Qiyu-Jiang mentioned this pull request

[fix] Undo replace_with_jit in unit tests #1107

Closed

Contributor

facebook-github-bot commented Sep 23, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Sep 23, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.


          Update on "[feat] Add ViT modules"

993bdba

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 24, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.


          Update on "[feat] Add ViT modules"

539b794

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Sep 29, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Sep 29, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Ryan-Qiyu-Jiang added a commit to Ryan-Qiyu-Jiang/mmf that referenced this pull request


          [feat] Add ViT modules

c5c38f9

Add modified huggingface ViT modules from megaVLT
supporting attention masks

ghstack-source-id: 1cc49c5bdec03b5538496fc9d7a7e02a86d87281
Pull Request resolved: facebookresearch#1099

Ryan-Qiyu-Jiang added a commit to Ryan-Qiyu-Jiang/mmf that referenced this pull request


          [feat] Add ViT modules

f3c6dac

Add modified huggingface ViT modules from megaVLT
supporting attention masks

ghstack-source-id: 1cc49c5bdec03b5538496fc9d7a7e02a86d87281
Pull Request resolved: facebookresearch#1099


          Update on "[feat] Add ViT modules"

f6e8656

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Ryan-Qiyu-Jiang mentioned this pull request

[feat] Add support for ViLT cmd pretraining #1113

Closed

Contributor

facebook-github-bot commented Oct 1, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Oct 1, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Ryan-Qiyu-Jiang added a commit to Ryan-Qiyu-Jiang/mmf that referenced this pull request


          [feat] Add ViT modules

421a74b

Add modified huggingface ViT modules from megaVLT
supporting attention masks

ghstack-source-id: a2c4616cd212dfde5903cd2188e1e5f4f61060ea
Pull Request resolved: facebookresearch#1099


          Update on "[feat] Add ViT modules"

d5d8cbe

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]


          Update on "[feat] Add ViT modules"

a118f51

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 8, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.


          Update on "[feat] Add ViT modules"

085f0b6

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 8, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Oct 8, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.


          Update on "[feat] Add ViT modules"

5b458ce

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 18, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Oct 19, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.


          Update on "[feat] Add ViT modules"

2b78a49

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 19, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.


          Update on "[feat] Add ViT modules"

2ef99ec

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 20, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Oct 20, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.


          Update on "[feat] Add ViT modules"

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 22, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Oct 22, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.


          Update on "[feat] Add ViT modules"

34a7b10

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 25, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Oct 25, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.


          Update on "[feat] Add ViT modules"

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 27, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.


          Update on "[feat] Add ViT modules"

66e05c3

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Differential Revision: [D31117429](https://our.internmc.facebook.com/intern/diff/D31117429)

[ghstack-poisoned]

Contributor

facebook-github-bot commented Oct 29, 2021

@Ryan-Qiyu-Jiang has updated the pull request. You must reimport the pull request before landing.

Contributor Author

Ryan-Qiyu-Jiang commented Oct 29, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

1 similar comment

Contributor Author

Ryan-Qiyu-Jiang commented Nov 1, 2021

@Ryan-Qiyu-Jiang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

This was referenced Nov 2, 2021

[feat] Add HeadsDict transformer head combiner #1141

Closed

[docs] Add ViLT model to website #1142

Closed

facebook-github-bot closed this

facebook-github-bot pushed a commit that referenced this pull request


          [feat] Add ViT modules (#1099)

c8c3d6d

Summary:
Pull Request resolved: #1099

Add modified huggingface ViT modules from megaVLT
supporting attention masks

Test Plan:
**Unit Test.**
Tests ViTModel, which uses all ViT submodules. Test checks if ViTModel can successfully perform a forward pass.

Unit test is skipped in older transformer versions, preserving backwards compatibility for unit tests.

**Modules used in ViLT Training**
ViT modules are used in the ViLT model which successfully performs pretraining and finetuning, ViLT model tested using mmf_run on VQA2.

**Tested Backwards Compatibility**
ViT modules backward compatibility with previous versions of transformers tested directly with
* 3.4.0 (throws helpful upgrade transformers version for ViT related modules)
* 4.5.0 (__spec missing error patched in previous transformers version bump), and
* 4.10.1

Reviewed By: ebsmothers

Differential Revision: D31117429

Pulled By: Ryan-Qiyu-Jiang

fbshipit-source-id: 9a473eb5d7e0183711d8d19dcd8f2b77b7792e03

facebook-github-bot deleted the gh/ryan-qiyu-jiang/4/head branch

December 3, 2021 15:15

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment