inpainting/masking question

[vrnord]

I am totally blown away by what this 3D conversion can do for 2D movies! The only drag at the moment is inpainting - so far everything I have tried just results in warped-looking gap fill. I tried the February Aether Medium model too but it made the infill look really flickery - like it wasn't staying temporally consistent over frames. So basically a couple questions:

1. what is the best inpaint method right now?
2. is there anything that can do something like Propainter to infill gaps, or is it something that can be looked into? https://github.com/sczhou/ProPainter
   https://github.com/daniabib/ComfyUI_ProPainter_Nodes
3. or, is there a way to output a side by side video with the gaps left blank (make all the gaps be in the right eye) and also output a separate mask video (the entire right eye frame black except the gaps that require infill are white) because then potentially I could run the video and mask through Propainter separately to generate background infill and then composite the result into the right eye view of the I3W-outputted video?

[nagadomi]

The warped-looking gap fill seen with inpainting is often caused by Edge Dilation. Setting it to 0 should reduce it significantly, but it may introduce other artifacts. See #484 .
My guess is that there is an alignment issue between depth boundaries and image edges when a low resolution depth map is used together with a high resolution RGB image.

---

If you want to separate the warping and inpainting steps explicitly and are willing to spend more time on inpainting quality, you may want to try Enoky's fork of StereoCrafter. I suspect people interested in workflows like yours may prefer diffusion based inpainting methods.
M2SVid and StereoPilot has also been released, so it may be worth looking for forks of that projects as well.
At the moment, I do not support these approaches, so you may be better off looking elsewhere.

[QuickscopingFTW]

i saw you testing some diffusion models in another of the tools in this project, do you think its still too heavy/complex to use for infill? what does the current inpainting method that is in iw3 use at the moment? would there be any way to improve that or would it require something totally different? ive tried the stereocrafter fork and m2svid and while they can end up with nice results it always ends up taking ages to set everything up to get it running for each video individually, thats why i love using iw3 as it is the best and simplest way to do things at the moment

[nagadomi]

Yes. If anything, the hardware requirements have been increasing.