Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VideoMAE] TensorFlow implementation #20351

Closed
wants to merge 24 commits into from
Closed

[VideoMAE] TensorFlow implementation #20351

wants to merge 24 commits into from
+1,769 −6

Conversation

sayakpaul
Copy link
Member

@sayakpaul sayakpaul commented Nov 21, 2022 •
edited
Loading

Closes #18641

TODO

  • modeling_tf_videomae.py
  • integration tests
  • rest of the tests
  • documentation

Sorry, something went wrong.

@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@sayakpaul
Copy link
Member Author

sayakpaul commented Nov 21, 2022 •
edited
Loading

I am hitting an issue and thought of asking for help.

The issue stems from the port of VideoMAESelfAttention i.e., TFVideoMAESelfAttention. Upon analyzing deeper, I think it's because of the mismatch between nn.functional.linear and my custom linear_transformation(). This is my investigative notebook where I have tried debugging the issue but still no luck so far. The assertion errors seem to be within low tolerance but can easily sum up.

Cc: @amyeroberts @gante @Rocketknight1

@sayakpaul sayakpaul self-assigned this Nov 21, 2022
sayakpaul
sayakpaul commented Nov 22, 2022
View reviewed changes
src/transformers/models/videomae/modeling_tf_videomae.py
return tf.transpose(tensor, perm=[0, 2, 1, 3])

def linear_transformation(self, inputs, weight, bias=None):
# weight = tf.transpose(weight) # To match the PT weights (particularly for cross-loading).
Copy link
Member Author

@sayakpaul sayakpaul Nov 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gante @ydshieh it didn't help btw.

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot closed this Feb 6, 2023
@amyeroberts amyeroberts reopened this Feb 7, 2023
@amyeroberts amyeroberts added the WIP Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress label Feb 7, 2023
@amyeroberts
Copy link
Collaborator

Added WIP label to prevent PR from being closed

@sayakpaul
Copy link
Member Author

No point in extending it further.

@sayakpaul sayakpaul closed this Jan 12, 2024
@sayakpaul sayakpaul deleted the feat/videomaetf branch January 12, 2024 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@sayakpaul sayakpaul

Labels
WIP Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add TF VideoMAE
3 participants
@sayakpaul @HuggingFaceDocBuilderDev @amyeroberts