Some question about the second stage

[mengVV]

Hi! I love your idea that improved ip2p in the second stage, but I don't really understand why you need to use the edited results to optimize the former Neural Video Field, and can you further explain that? Did you do the ablations about the the additional optimization of the NVF in the second stage?

Looking forward to your answer.

[Ysz2022]

I guess you want to ask that why I use the edited frame to optimize NVF rather than directly adopting the edited frames as the edited results right?

That is because the edited frames generated by IP2P+ is not temporally consistent and cannot be concated to a coherent video.

I cannot understant your last question: Did you do the ablations about the the additional optimization of the NVF in the second stage?. Maybe you can further explain it in more detail :)

[mengVV]

I guess you want to ask that why I use the edited frame to optimize NVF rather than directly adopting the edited frames as the edited results right?

That is because the edited frames generated by IP2P+ is not temporally consistent and cannot be concated to a coherent video.

I cannot understant your last question: Did you do the ablations about the the additional optimization of the NVF in the second stage?. Maybe you can further explain it in more detail :)

Hi! Thanks for your attention, and maybe I can clarify my question:

1. At the inference you only use NVF, but how can you inject the instruction/prompt/text information for editing?
2. As for my understanding, NVF could reconstruct the video, and why you input the rendered frame to IP2P+?

[Ysz2022]

Notice that NVF itself does not have any capability of editing, we impart editing effect in the second stage (field editing stage) of training.

1、In the second stage, we have optimized NVF by edited frames, which are generated based on the prompt information. So we can directly use NVF to render the video with the desired editing effect after training.

2、IP2P+ is required to edit the rendered frames in the second stage. Our editing effect mainly depends on it.

Maybe you can read the paragraph of "Field editing stage" in Sec. 3.2 of our paper :)

Thanks for your attention on our work, if you have any questions, please tell me and I will try my best to answer.

[mengVV]

So when inferencing you will input the instruction/prompt/text to the NVF, and dont need IP2P+? Or for different videos, you need to optimize the NVF respectively?

[Ysz2022]

During inference, there is no need to input any instruction to NVF. We only input the coordinates and get pixels, as shown in the Eq. 6 of our paper.

For different videos, we need to retrain NVF :)

[Ysz2022]

Excuse me :) May I ask if my response has solved your questions?

[mengVV]

Yes I got it, and so nice of u. Sorry for replying late cause I was on a business trip.

[Ysz2022]

I will temporally close this issue and you are welcome to reopen it if you need :)