Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding swin to the self-supervised swav #974

Open
AhmedHamdi101 opened this issue Feb 7, 2023 · 2 comments 路 May be fixed by #975
Open

Adding swin to the self-supervised swav #974

AhmedHamdi101 opened this issue Feb 7, 2023 · 2 comments 路 May be fixed by #975
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@AhmedHamdi101
Copy link

馃殌 Feature

Addition of Swin, SwinV2 to the swav self-supervised models

Motivation and Pitch

Swin, SwinV2 vision transformers are currently one of the top-performing models on multiple computer vision tasks, but such as all transformers, training them would need a large number of samples.
I thought that a self-supervised approach such as SWAV would benefit those models and help use those vision transformers in low resources tasks.

I am very eager to add it myself.
@AhmedHamdi101 AhmedHamdi101 added enhancement New feature or request help wanted Extra attention is needed labels Feb 7, 2023
@AhmedHamdi101 AhmedHamdi101 linked a pull request Feb 7, 2023 that will close this issue
Open
8 tasks
@Atharva-Phatak
Copy link
Contributor

SWAV is already implemented in lightning bolts. You will have to inherent SWAV class and override init_model method for training SWIN transformers. FYI I recommend you check DINO for training vision transformers. SWaV is not the best candidate.

@AhmedHamdi101
Copy link
Author

SWAV is already implemented in lightning bolts. You will have to inherent SWAV class and override init_model method for training SWIN transformers. FYI I recommend you check DINO for training vision transformers. SWaV is not the best candidate.

But when I override the init_model I will also need to modify SWIN architecture by

  • Splitting the default forward function into forward_backbone and forward_head
  • Add a new forward which calls both of them
  • Add prototype layer and conditions for projection head in the model init

So I added the modified SWIN in swin_swav.py and instead of overriding the init_model, I just added them as arch options to be chosen for ease of use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

added swin swav model AhmedHamdi101/lightning-bolts
2 participants
@Atharva-Phatak @AhmedHamdi101