-
-
Notifications
You must be signed in to change notification settings - Fork 948
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Keypoints not properly transformed through Augmentations #2684
Comments
Not all augmentations have support for keypoints yet (see #941), contributions are welcomed. @shijianjian do we have somewhere a list with what augmentations support each case? |
In theory, all the augmentations inherits from the |
Upon digging a little deeper, I think I figured out the issue which is pretty simple albeit unintuitive for an end-user. Keypoints can be transformed to be moved out of the image but then moved back into the image by subsequent transformations [e.g., Translate] with zeros for padding. The end result of this is that keypoints become detached from their original content as seen below. If you have a series of augmentations as is common, it doesn't seem like there's a simple way to determine if a keypoint is no longer a valid one; perhaps returning a mask to denote which ones are valid or setting invalid keypoints to a special value [-1, NaN] would be a solution. The example below is with |
Ah. Yes, I remember the design now. Since we use the same point transformation for bounding boxes, we do not remove those invalid keypoints. Those need to be kept otherwise the boxes will not have four corners. I would vote for having a |
geometrically speaking the keyword pts = [....]
pts_filtered = [p for pts if p.valid] besidesi, i'd also consider expanding the kornia/kornia/geometry/vector.py Line 95 in 40d07ec
|
I don't think it should be very difficult but I'm not very familiar with Kornia (this is my first time using it actually). I added a basic implementation but I'm not sure how robust it is and whether it works for all geometric augmentations. The few I tested (Shear, Translate, Crop, Rotate, Flip) seem to work though. It's a little hacky as it checks the I think a mask is preferable to making each point into an object with a field, at least for my use-case with a very dense grid of key points. It also might make sense to set the masked out key points to some special value so it's clear to the user that they need to ignore these values, unless there's a use case for them. On the surface, key points are ostensibly used for correspondence so it's hard to see why someone would want this broken. |
link to #2689 |
Describe the bug
Some combinations of augmentation operations cause keypoints to be incorrectly transformed. A demo script (below) creates a white image and a grid of keypoints. In the output image, all keypoints should be on the white input, but often they are not.
It seems to happen when a translate augmentation is applied along with other augmentations [cropping, rotation, etc.] but also happens with, for example, RandomResizedCrop + RandomRotation.
Reproduction steps
The text was updated successfully, but these errors were encountered: