-
Notifications
You must be signed in to change notification settings - Fork 25.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ViTPose #30530
base: main
Are you sure you want to change the base?
Add ViTPose #30530
Conversation
@NielsRogge Hi Niels! Does this current PR works properly? (I want to do some test for this) |
Thanks for working on this @NielsRogge! I can see there's a few bits unfinished e.g. tests. Is there a particular bit of code you'd like me to look at for a maintainers perspective? For the issue description, there's a related model request is here: #24915 |
@NielsRogge I'm unsubscribing atm, so that I don't get notifications on every new push. You just need to ping me again with my username when it's ready for review and I'll get notified |
It would be great to have a first round of review as the PR is in a ready state. @amyeroberts |
|
||
|
||
name_to_path = { | ||
"vitpose-base-simple": "/Users/nielsrogge/Documents/ViTPose/vitpose-b-simple.pth", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"vitpose-base-simple": "/Users/nielsrogge/Documents/ViTPose/vitpose-b-simple.pth", | |
"vitpose-base-simple": "https:\/\/4mjpca.sn.files.1drv.com\/y4mip6jbupeZ3YzICoNJYUb6yGEheWXkicKj0tvp1Sfq8BztlH8ieD63z2ZRYiTBzvDxKXFqd_wa5m8NHnBsURmpClZySMSJjS3hxrU2bFArawJ5mAVZsni4LmsfWs_K1dnIzDumXXuanSopYKm0O-Bx5z4JerIfGoE6riAtY_ni5_paFl46jGTE82U8J10Cm3gxHv2DSfOkrgV7SkmUKvnjg\/vitpose-b-simple.pth?download&psid=1", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
import requests
def download_file(url, local_filename):
# Sending requests with stream=True allows downloading large files
with requests.get(url, stream=True) as response:
response.raise_for_status() # Raise an exception if the request fails
with open(local_filename, 'wb') as file:
# Efficiently download large files by iterating over content in chunks
for chunk in response.iter_content(chunk_size=8192):
file.write(chunk)
return local_filename
# Given URL and the filename to save
url = "https://4mjpca.sn.files.1drv.com/y4mip6jbupeZ3YzICoNJYUb6yGEheWXkicKj0tvp1Sfq8BztlH8ieD63z2ZRYiTBzvDxKXFqd_wa5m8NHnBsURmpClZySMSJjS3hxrU2bFArawJ5mAVZsni4LmsfWs_K1dnIzDumXXuanSopYKm0O-Bx5z4JerIfGoE6riAtY_ni5_paFl46jGTE82U8J10Cm3gxHv2DSfOkrgV7SkmUKvnjg/vitpose-b-simple.pth?download&psid=1"
local_filename = "vitpose-b-simple.pth"
# Execute file download
download_file(url, local_filename)
print(f"File downloaded as {local_filename}")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can convert this code with the cloud uploaded weight!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding this model!
I've only done a first high-level pass. Normally I'd ask for the backbone to be added in a separate PR, but as the modeling files are relatively small, I think it's OK.
Main comments are about the image processing: poat-processing should take and return torch tensors; there should be more tests to make sure the pre and post processing work on batched inputs and outputs are as expected, particularly for the custom transforms; cv2 logic should be removed
if is_cv2_available(): | ||
# TODO get rid of cv2? | ||
import cv2 | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
cv2_image = ( | ||
image | ||
if input_data_format == ChannelDimension.LAST | ||
else to_channel_dimension_format(image, ChannelDimension.LAST, input_data_format) | ||
) | ||
image = cv2.warpAffine(cv2_image, transformation, size, flags=cv2.INTER_LINEAR) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All cv2 logic should be removed
elif width < aspect_ratio * height: | ||
width = height * aspect_ratio | ||
|
||
# pixel std is 200.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Always?
What does this PR do?
This PR adds ViTPose as introduced in ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation.
Here's a demo notebook - note that the API might change: https://colab.research.google.com/drive/15_3gjcC0wtKSH85k76zewt81eUJIEWWA?usp=sharing.
To do: