Recurrent Modulation Unit in RM-Depth (CVPR2022) vs Recurrent Multi-Scale Feature Modulation in R-MSFM (ICCV21)

[jsczzzk]

Congratulations on your new work!

I think the idea of iteratively refining the fusion using Recurrent Modulation Unit in RM-Depth is similar to the Recurrent Multi-Scale Feature Modulation in R-MSFM. Both of them use gated recurrent unit (GRU) to fuse the features . However, R-MSFM was not cited in RM-Depth.

[twhui]

It seems to me that your ICCV 2021 paper was released to the public just a few weeks before the paper registration deadline of CVPR 2022. Therefore, I did not aware of your work when I was performing the literature review.

The technical details of RM-Depth are different from R-MSFM. Particularly, GRU is not used in RM-Depth.

Your depth CNN is basically a RAFT-based approach. Your MSFM uses GRU as RAFT to "modulate" the content between previous hidden state and encoder feature at a fixed 1/8 scale (In fact, I do not understand why you used the word "modulate" here. Maybe, "fuse" or "combine" is better).

My depth CNN uses RMU for dynamic and iterative feature fusion between encoder and decoder at several scales (Fig. 2b). Each RMU consists of two phases: modulation and update (Fig. 3). Feature from the depth encoder is modulated according to the previous hidden state through an adaptive modulation function (Eqs. 4a, 4b). Then, the new hidden state is updated as a weighted combination between the previous hidden state and the modulated encoder feature (Eqs. 5a, 5b). RMU is different from GRU. GRU does not have modulation phase. Furthermore, GRU uses two gates while RMU uses one gate.

In fact, I also proposed to use GRU for depth refinement as yours to CVPR 2021 (even earlier than yours by 4 months). However, the paper was not accepted.

[twhui]

Did you use my posted issue in haofeixu/gmflow#7 as a template? Interestingly, the style of writing of your posted issue is the same as me :)

[jsczzzk]

Hi，thanks for the clarification. Your careful explanations can help readers better understand both our methods. I carefully read your paper and think this is an excellent work in recurrent depth estimation. In addition, I have read your liteflownet before. This work is also a wonderful work. Looking forward to your further work.

Best wishes,
Zhongkai