Add example for optical flow visualizaition and RAFT #5316

oke-aditya · 2022-01-29T18:17:35Z

Closes #5309

Thanks a lot to @NicolasHug for taking it over.

…_raft

facebook-github-bot · 2022-01-29T18:17:41Z

💊 CI failures summary and remediations

As of commit 0352a47 (more details on the Dr. CI page):

✅ None of the CI failures appear to be your fault 💚

1/1 broken upstream at merge base 74a1efc since Feb 04

🚧 1 ongoing upstream failure:

These were probably caused by upstream breakages that are not fixed yet.

binary_win_wheel_py3.9_cu115 since Feb 04 (22f8dc4)
- 🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

NicolasHug

Thanks @oke-aditya this looks great so far! Made a quick review as discussed offline

gallery/plot_optical_flow.py

NicolasHug · 2022-02-02T15:22:59Z

gallery/plot_optical_flow.py

+from torchvision.utils import flow_to_image
+img = flow_to_image(flow)
+
+img = img.squeeze(0)


It'd be nice to show the flow image and frame_1 side by side :)

Exactly wanted to do that :( Faced some errors, will try again.

NicolasHug · 2022-02-02T15:23:56Z

gallery/plot_optical_flow.py

+
+"""
+
+# sphinx_gallery_thumbnail_path = "../../gallery/assets/optical_flow_thumbnail.png"


We might need to remove the thumbnail, I'm not sure we have the rights of the optical_flow_thumbnail.png image that was uploaded. But I'm sure we can use on of the images that this example will generate!

Yes it's pending. Haven't worked much on this yet, will try to complete over this week.

…_raft

oke-aditya · 2022-02-06T18:10:41Z

Not much well over the weekend 😢 will try to finish this in few days

NicolasHug · 2022-02-07T15:51:50Z

@oke-aditya thank you so much for your work so far.

I took care of polishing the example a bit, here are the latest rendered docs: https://1177465-73328905-gh.circle-artifacts.com/0/docs/auto_examples/plot_optical_flow.html#sphx-glr-auto-examples-plot-optical-flow-py

LMK what you think of it!

gallery/plot_optical_flow.py

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

oke-aditya

Looks fantastic, awesome. Few comments ❤️

oke-aditya · 2022-02-07T16:51:44Z

gallery/plot_optical_flow.py

+# GIF using ffmpeg with e.g.:
+#
+# `
+# ffmpeg -f image2 -framerate 30 -i predicted_flow_%d.jpg -loop -1 flow.gif


I guess we can put the rendered GIF here? (If sphinx can render it)

I'm not sure sphinx can. It's a good idea though, I'll do it once I find a way to reduce the size of the gif to a decent size (the ones i have right now are 300MB...)

Looks like we can.
https://gifs-as-documentation.readthedocs.io/en/latest/

We need to compress the GIF though :( Maybe possible by running in very small video size say 120x60? For 3 seconds?

gallery/plot_optical_flow.py

oke-aditya · 2022-02-07T16:59:03Z

gallery/plot_optical_flow.py

+
+
+plt.rcParams["savefig.bbox"] = "tight"
+# sphinx_gallery_thumbnail_number = 2


I couldn't find the thumbnail. You, sure it's there?

Yes this tells sphinx-gallery to use the second image as the thumbnail instead of the first one: https://1177465-73328905-gh.circle-artifacts.com/0/docs/generated/torchvision.models.optical_flow.raft_large.html#torchvision.models.optical_flow.raft_large

gallery/plot_optical_flow.py

oke-aditya · 2022-02-07T17:42:03Z

gallery/plot_optical_flow.py

+# [-1, 1]. The frames we got from :func:`~torchvision.io.read_video` are int
+# images with values in [0, 255], so we will have to pre-process them. We also
+# reduce the image sizes for the example to run faster. Image dimension must be
+# divisible by 8.


Why do the image dimensions need to be divisble by 8?

It's a hardcoded constraint within the model, the feature extractor downsamples the images by 8

My doubt is it hard necessicity that image sizes should be divisible by 8. Or its adjusted by the model.
The text description above says. Image sizes must be divisible by 8. Meaning that the model does not adjust.
Clearer way can be. Image sizes divisible by 8 are processed faster.

It's really a hardcoded constraint and the model cannot accept images that aren't divisible by 8 (even if it wanted to):

vision/torchvision/models/optical_flow/raft.py

Line 456 in 74a1efc

torch._assert((h % 8 == 0) and (w % 8 == 0), "input image H and W should be divisible by 8")

It has to be exactly divisible by 8 because we first downsample the inputs by 8, predict a downsampled flow, and then upsample the predicted flow by a factor of 8:

vision/torchvision/models/optical_flow/_utils.py

Lines 26 to 32 in 74a1efc

def upsample_flow(flow, up_mask: Optional[Tensor] = None):

"""Upsample flow by a factor of 8.

If up_mask is None we just interpolate.

If up_mask is specified, we upsample using a convex combination of its weights. See paper page 8 and appendix B.

Note that in appendix B the picture assumes a downsample factor of 4 instead of 8.

"""

If the image wasn't a multiple of 8 to begin with, we wouldn't be able to upsample the flow to the right dimensions.

The fact that it's 8 is somewhat arbitrary (we could downsample by 4 and upsample by 4) but a) this would not follow the paper and b) we would still require images to be divisible by an integer N in general.

Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com>

This reverts commit 2c7a468.

NicolasHug

Thanks a lot for your help @oke-aditya , LGTM, I'll merge when the docs are built!

NicolasHug · 2022-02-09T09:55:06Z

Looks good lessgooooo https://1182173-73328905-gh.circle-artifacts.com/0/docs/auto_examples/plot_optical_flow.html#sphx-glr-auto-examples-plot-optical-flow-py

oke-aditya · 2022-02-09T10:03:32Z

Lessssss goooooooooooooooo. Yay

Summary: * Start adding example * Add thumbnail and text * Replace video * Improve * Change default weights of RAFT model builders * WIP * WIP * update handle_legacy_interface input * lots of stuff * Oops, wrong default * Typo * NITs * Reduce image size * Update gallery/plot_optical_flow.py * Remove link to profile * Update gallery/plot_optical_flow.py * Address comments * Nits * Revert "Remove link to profile" This reverts commit 2c7a468. Reviewed By: NicolasHug Differential Revision: D34140253 fbshipit-source-id: e3a129d641335a38ac5c5e3299824e1794c7bb52 Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com> Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com> Co-authored-by: Nicolas Hug <nicolashug@fb.com> Co-authored-by: Nicolas Hug <contact@nicolas-hug.com> Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

oke-aditya added 2 commits January 29, 2022 23:46

Start adding example

355ae63

Merge branch 'main' of https://github.com/pytorch/vision into gallery…

17364b4

…_raft

pytorch-bot bot added the ciflow/default label Jan 29, 2022

facebook-github-bot added the cla signed label Jan 29, 2022

oke-aditya added 2 commits January 31, 2022 00:40

Add thumbnail and text

d8a7237

Replace video

1948937

NicolasHug reviewed Feb 2, 2022

View reviewed changes

oke-aditya added 2 commits February 6, 2022 23:38

Improve

68d6795

Merge branch 'main' of https://github.com/pytorch/vision into gallery…

ff92efe

…_raft

NicolasHug added 11 commits February 7, 2022 11:03

Change default weights of RAFT model builders

f2ab8a3

Merge branch 'raft_default_weights' into gallery_raft

7bc3482

WIP

dd8dc34

WIP

7c6a550

update handle_legacy_interface input

d1feb73

lots of stuff

943a1dc

Oops, wrong default

d786cf8

Merge branch 'raft_default_weights' into gallery_raft

ffa8212

Typo

2bd7f93

NITs

f28bf56

Reduce image size

ee55699

Merge branch 'main' of github.com:pytorch/vision into gallery_raft

1c171f5

NicolasHug added enhancement module: documentation labels Feb 7, 2022

NicolasHug marked this pull request as ready for review February 7, 2022 16:02

datumbox reviewed Feb 7, 2022

View reviewed changes

gallery/plot_optical_flow.py Outdated Show resolved Hide resolved

Update gallery/plot_optical_flow.py

4322dad

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

oke-aditya commented Feb 7, 2022

View reviewed changes

oke-aditya requested a review from NicolasHug February 7, 2022 17:43

NicolasHug and others added 7 commits February 8, 2022 09:37

Merge branch 'main' of github.com:pytorch/vision into gallery_raft

c9c7761

Remove link to profile

2c7a468

Update gallery/plot_optical_flow.py

628b993

Co-authored-by: Aditya Oke <47158509+oke-aditya@users.noreply.github.com>

Address comments

dd7e1a8

Nits

2baa55d

Revert "Remove link to profile"

e305f16

This reverts commit 2c7a468.

Merge branch 'main' into gallery_raft

0352a47

NicolasHug approved these changes Feb 9, 2022

View reviewed changes

NicolasHug merged commit c39c23e into pytorch:main Feb 9, 2022

oke-aditya deleted the gallery_raft branch February 9, 2022 10:03


		"""

		# sphinx_gallery_thumbnail_path = "../../gallery/assets/optical_flow_thumbnail.png"



		plt.rcParams["savefig.bbox"] = "tight"
		# sphinx_gallery_thumbnail_number = 2

	def upsample_flow(flow, up_mask: Optional[Tensor] = None):
	"""Upsample flow by a factor of 8.

	If up_mask is None we just interpolate.
	If up_mask is specified, we upsample using a convex combination of its weights. See paper page 8 and appendix B.
	Note that in appendix B the picture assumes a downsample factor of 4 instead of 8.
	"""

Add example for optical flow visualizaition and RAFT #5316

Add example for optical flow visualizaition and RAFT #5316

Uh oh!

Conversation

oke-aditya commented Jan 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Jan 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🚧 1 ongoing upstream failure:

Uh oh!

NicolasHug left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oke-aditya commented Feb 6, 2022

Uh oh!

NicolasHug commented Feb 7, 2022

Uh oh!

Uh oh!

oke-aditya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oke-aditya Feb 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NicolasHug Feb 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NicolasHug left a comment

Choose a reason for hiding this comment

Uh oh!

NicolasHug commented Feb 9, 2022

Uh oh!

oke-aditya commented Feb 9, 2022

Uh oh!

Uh oh!

oke-aditya commented Jan 29, 2022 •

edited

Loading

facebook-github-bot commented Jan 29, 2022 •

edited

Loading

oke-aditya Feb 8, 2022 •

edited

Loading

NicolasHug Feb 8, 2022 •

edited

Loading