Processing images from video

[bhack]

Is there any specific setting to take advantage of images from video sequences?

[jytime]

Hi @bhack we are preparing for this and should be available in next version.

[bhack]

Do you have just another planned version or paper?

[jytime]

yes we are planning to release a new version of (a) a hugging face demo that can run on arbitrary input videos and (b) training script. The new version will also reduce the gpu usage by half

[bhack]

Do you will have also an option to mask non rigid pixels in video mode?

[bhack]

E.g. like the clustering in https://littlepure2333.github.io/GFlow/
or https://www.cis.upenn.edu/~leijh/projects/mosca/

[bhack]

@jytime So is there something available in the new V2?

[bhack]

In the case currently we cannot autocluster/classify rigid and non rigid points can you at least support an init mask on the dynamic parts in the tracker?

[jytime]

Hey,

Currently, we do not have an update on the video processing feature as we are focusing on improving memory efficiency, as the mem problem seems to bother many people.

Regarding non-rigid pixels, you have two options:

(a) Avoid using non-rigid points as query points. You can achieve this by passing a mask to the get_query_points() function

vggsfm/demo.py

Line 453 in 8c47df8

def get_query_points(query_image, query_method, max_query_num=4096, det_thres=0.005):

or

(b) set all non-rigid points as non-visible, such as done at:

vggsfm/demo.py

Line 236 in 8c47df8

# If necessary, force all the predictions at the padding areas as non-visible

All the non-visible will be ignored during triangulation.

[bhack]

What kind of mask density this is going to require over a sequence? How many query you are preparing?

[cv-lab-x]

any schedule to release the version of processing video ?
Looking forward to your reply, thanks.
@jytime

[jytime]

Hi I am doing it now. I am comparing different methods to filter out dynamic objects in the videos. One potential solution is something like @bhack mentioned above. I am also comparing the off-the-shelf motion segmentation methods. Let's see which will win.

[bhack]

Yes I've shared other fresh resources and comments in the other issue #24 (comment)

[jytime]

Thanks for sharing these all. A lot of distractions recently. Hope I can do my best to share it as soon as possible.