Add the iterative Lucas-Kanade (iLK) optical flow method #4161

rfezzani · 2019-09-13T15:57:33Z

Description

Here is an implementation of the iterative Lucas-Kanade (iLK) method for optical flow estimation [1].
The Lucas-Kanade method is a local approach for the estimation of optical flow (as opposed to global approaches such TV-L1 for example). It is fast, robust and requires basically only one parameter that is the size of the neighborhood considered for the estimation.

[1]: Le Besnerais, G., & Champagnat, F. (2005, September). Dense optical flow by iterative local window registration. In IEEE International Conference on Image Processing 2005 (Vol. 1, pp. I-137). IEEE.

Checklist

Docstrings for all functions
Gallery example in ./doc/examples (new features only)
Benchmark in ./benchmarks, if your changes aren't covered by an
existing benchmark
Unit tests
Clean style in the spirit of PEP8

For reviewers

Check that the PR title is short, concise, and will make sense 1 year
later.
Check that new functions are imported in corresponding __init__.py.
Check that new features, API changes, and deprecations are mentioned in
doc/release/release_dev.rst.
Consider backporting the PR with @meeseeksdev backport to v0.14.x

pep8speaks · 2019-09-13T15:57:36Z

Hello @rfezzani! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file doc/examples/registration/plot_opticalflow.py:

Line 114:14: E226 missing whitespace around arithmetic operator
Line 114:24: E226 missing whitespace around arithmetic operator

Comment last updated at 2020-10-02 08:50:49 UTC

jni · 2019-09-15T14:29:50Z

Ha! I didn't realise the question of how to present multiple optical flow algorithms in the API would become so relevant so soon! 😂 Thanks @rfezzani! I also notice that it's nD out of the box, a very nice touch. ;)

I don't have time for a deep review right this second but it looks pretty great on first glance!

rfezzani · 2019-09-15T21:44:52Z

😂 Thank you @jni, the code was almost done in pyimof, I just had to implement the nD support ;-) I hope you like it.
I had a look on Travis CI, but I don't understand what makes it Fail :-(

…into Add_ilk_optical_flow_method

lagru · 2019-09-25T22:52:10Z

I had a look on Travis CI, but I don't understand what makes it Fail :-(

Both failures are due to optical_flow_ilk returning float64 when float32 is expected. Does that help or did you mean you don't understand why that function returns the wrong dtype?

lagru · 2019-09-25T22:56:58Z

skimage/registration/tests/test_ilk.py

+    # Estimate the flow at double precision
+    flow_f64 = optical_flow_ilk(image0, image1, dtype='float64')
+
+    assert flow_f64.dtype == 'float64'


First time seeing that way to check dtypes. Seems a bit brittle and doesn't seem to work for cases such as np.float64 == "float64" which will return False. So maybe use np.float64 here (and similar at other places) instead?

You are right, np.float64 == "float64" returns False, but np.dtype(np.float64) == 'float64' is True. Here I compare data type object...
But I will check if this can be done differently ;-)

flow.dtype == np.float64 should work?

Yes, absolutely :-)

skimage/registration/tests/test_ilk.py

Co-Authored-By: Lars Grueter <lagru@users.noreply.github.com>

rfezzani · 2019-09-26T07:38:30Z

I had a look on Travis CI, but I don't understand what makes it Fail :-(

Both failures are due to optical_flow_ilk returning float64 when float32 is expected. Does that help or did you mean you don't understand why that function returns the wrong dtype?

I actually don't understand why that function returns the wrong dtype 😄. Since the tests pass in the other CI checks...

rfezzani · 2019-09-26T08:52:57Z

OK, I finally reproduced the CI failure: It comes from numpy version 1.13.3!

rfezzani · 2019-09-26T10:04:05Z

The CI failure is related to the issue #4189.

…into Add_ilk_optical_flow_method

rfezzani · 2020-09-23T15:33:22Z

Thank you very much @grlee77 for your review 😉

skimage/registration/_optical_flow.py

rfezzani · 2020-09-25T07:16:42Z

Thank you again @grlee77 for your excellent review, I think I addressed all your requests now 😉.

grlee77

Thanks @rfezzani for addressing the comments. This seems good to go.

emmanuelle

just a few comments as I was reading

doc/examples/registration/plot_opticalflow.py

skimage/registration/_optical_flow.py

Co-authored-by: Emmanuelle Gouillart <emma@plot.ly>

emmanuelle · 2020-09-28T07:10:59Z

@rfezzani one thing which I've been missing either in docstrings or in the gallery example (maybe both) is when to use which optical flow algorithm. From what I understand, one should use TV-l1 to allow for discontinuities in the flow, but when the flow is smooth one should prefer ilk which is faster. Is this correct? Could you please add a few explanations on how to select an optical flow algorithm? Algorithm selection is maybe as hard as parameter selection for our users :-).

emmanuelle · 2020-09-28T08:15:25Z

@rfezzani thank you very much for the PR! I have finished reading through the Plyer et al. paper, and the PR is ready to go for me. The only caveat would be to add a few explanations about when to use which algorithm, please tell us if you have the time to add it now or not. But there should not be other comments from me!

rfezzani · 2020-09-29T06:56:15Z

Sorry @emmanuelle, I didn't found time yesterday :/ I will try to manage adding notes in the doc today.

emmanuelle · 2020-09-29T07:07:15Z

Thanks @rfezzani :-). Still in time for the release, don't worry.

rfezzani · 2020-10-02T08:53:12Z

@emmanuelle, I modified optical_flow_ilk docstring's long description:

The iterative Lucas-Kanade (iLK) solver is applied at each level
of the image pyramid. iLK [1]_ is a fast and robust alternative to
TVL1 algorithm although less accurate for rendering flat surfaces
and object boundaries (see [2]_).

Is it enough or should I elaborate more?

emmanuelle · 2020-10-02T09:02:45Z

Thanks @rfezzani . Let's wait for CI and then I'll merge!

rfezzani · 2020-10-02T15:50:37Z

I think CI failure is not related...

emmanuelle · 2020-10-02T20:13:45Z

Merged :-) !

rfezzani · 2020-10-02T22:10:17Z

🎉 Thank you @emmanuelle !

Riadh added 2 commits September 13, 2019 17:47

Add iterative Lucas-Kanade (iLK) optical flow method

6beceac

Add tests for iLK optical flow method

04a5deb

Riadh and others added 6 commits September 13, 2019 18:04

Fix PEP8

df14ddc

Fix PEP8

9b97d68

Fix tests

6206163

Revert test_tvl1 to original state

2499d4a

Add benchmark for ilk optical flow

14c5f9a

Add small optimization

5baa3dc

sciunto added the type: new feature label Sep 17, 2019

Merge branch 'master' of https://github.com/scikit-image/scikit-image …

d31ecf6

…into Add_ilk_optical_flow_method

lagru reviewed Sep 25, 2019

View reviewed changes

rfezzani and others added 2 commits September 26, 2019 09:25

Update skimage/registration/tests/test_ilk.py

9990220

Co-Authored-By: Lars Grueter <lagru@users.noreply.github.com>

Update skimage/registration/tests/test_ilk.py

819c58d

Co-Authored-By: Lars Grueter <lagru@users.noreply.github.com>

rfezzani mentioned this pull request Sep 26, 2019

skimage.util.convert not converting to 'float32' with numpy v1.13.3 #4189

Closed

Riadh added 9 commits September 27, 2019 09:29

Make skimage imports relative

4823852

Fix typo

9e3e13d

Merge branch 'master' of https://github.com/scikit-image/scikit-image …

92d52c9

…into Add_ilk_optical_flow_method

Merge branch 'master' of https://github.com/scikit-image/scikit-image …

324cc48

…into Add_ilk_optical_flow_method

Merge branch 'master' of https://github.com/scikit-image/scikit-image …

ac6c62b

…into Add_ilk_optical_flow_method

Fix merge and add references' DOI

cb8d612

Merge branch 'master' of https://github.com/scikit-image/scikit-image …

8fff65a

…into Add_ilk_optical_flow_method

Merge branch 'master' of https://github.com/scikit-image/scikit-image …

aa1ed0c

…into Add_ilk_optical_flow_method

Merge branch 'master' of https://github.com/scikit-image/scikit-image …

85d5ad5

…into Add_ilk_optical_flow_method

grlee77 reviewed Sep 24, 2020

View reviewed changes

skimage/registration/_optical_flow.py Outdated Show resolved Hide resolved

Riadh added 3 commits September 25, 2020 09:04

Make gradient arrays stacking explicit

c9fe622

Rename the variable It to error_image

16ed5dc

Add ref about impemented algorithm

61ebe84

Add ref about impemented algorithm

77724fd

grlee77 approved these changes Sep 25, 2020

View reviewed changes

Registration and warping automation moved this from Under Review to Ready to Merge Sep 25, 2020

rfezzani added the action: mrg+1 label Sep 27, 2020

emmanuelle reviewed Sep 27, 2020

View reviewed changes

doc/examples/registration/plot_opticalflow.py Outdated Show resolved Hide resolved

doc/examples/registration/plot_opticalflow.py Outdated Show resolved Hide resolved

emmanuelle reviewed Sep 27, 2020

View reviewed changes

skimage/registration/_optical_flow.py Outdated Show resolved Hide resolved

rfezzani and others added 2 commits September 28, 2020 00:05

Update doc/examples/registration/plot_opticalflow.py

ec2f13b

Co-authored-by: Emmanuelle Gouillart <emma@plot.ly>

Update doc/examples/registration/plot_opticalflow.py

6c7a394

Co-authored-by: Emmanuelle Gouillart <emma@plot.ly>

Refactor axe swapping

97dffea

Add description of the differences between iLK and TVL1 results

d33a8f7

emmanuelle merged commit 769a555 into scikit-image:master Oct 2, 2020

Registration and warping automation moved this from Ready to Merge to Done Oct 2, 2020

grlee77 mentioned this pull request Jan 28, 2021

More consistent support for single precision computation across skimage #5205

Closed

grlee77 added 🙏 Feature request and removed type: new feature labels Feb 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the iterative Lucas-Kanade (iLK) optical flow method #4161

Add the iterative Lucas-Kanade (iLK) optical flow method #4161

rfezzani commented Sep 13, 2019 •

edited

pep8speaks commented Sep 13, 2019 •

edited

jni commented Sep 15, 2019

rfezzani commented Sep 15, 2019

lagru commented Sep 25, 2019

lagru Sep 25, 2019

rfezzani Sep 26, 2019

jni Feb 6, 2020

rfezzani Feb 6, 2020

rfezzani commented Sep 26, 2019 •

edited

rfezzani commented Sep 26, 2019

rfezzani commented Sep 26, 2019

rfezzani commented Sep 23, 2020

rfezzani commented Sep 25, 2020

grlee77 left a comment

emmanuelle left a comment

emmanuelle commented Sep 28, 2020

emmanuelle commented Sep 28, 2020

rfezzani commented Sep 29, 2020

emmanuelle commented Sep 29, 2020

rfezzani commented Oct 2, 2020

emmanuelle commented Oct 2, 2020

rfezzani commented Oct 2, 2020

emmanuelle commented Oct 2, 2020

rfezzani commented Oct 2, 2020

Add the iterative Lucas-Kanade (iLK) optical flow method #4161

Add the iterative Lucas-Kanade (iLK) optical flow method #4161

Conversation

rfezzani commented Sep 13, 2019 • edited

Description

Checklist

For reviewers

pep8speaks commented Sep 13, 2019 • edited

Comment last updated at 2020-10-02 08:50:49 UTC

jni commented Sep 15, 2019

rfezzani commented Sep 15, 2019

lagru commented Sep 25, 2019

lagru Sep 25, 2019

Choose a reason for hiding this comment

rfezzani Sep 26, 2019

Choose a reason for hiding this comment

jni Feb 6, 2020

Choose a reason for hiding this comment

rfezzani Feb 6, 2020

Choose a reason for hiding this comment

rfezzani commented Sep 26, 2019 • edited

rfezzani commented Sep 26, 2019

rfezzani commented Sep 26, 2019

rfezzani commented Sep 23, 2020

rfezzani commented Sep 25, 2020

grlee77 left a comment

Choose a reason for hiding this comment

emmanuelle left a comment

Choose a reason for hiding this comment

emmanuelle commented Sep 28, 2020

emmanuelle commented Sep 28, 2020

rfezzani commented Sep 29, 2020

emmanuelle commented Sep 29, 2020

rfezzani commented Oct 2, 2020

emmanuelle commented Oct 2, 2020

rfezzani commented Oct 2, 2020

emmanuelle commented Oct 2, 2020

rfezzani commented Oct 2, 2020

rfezzani commented Sep 13, 2019 •

edited

pep8speaks commented Sep 13, 2019 •

edited

rfezzani commented Sep 26, 2019 •

edited