How to find (shadow) relevant features in the latent space?

[SleyAI]

Hello, I'm currently trying to implement the first step of your proposed algorithm (input: portrait image, face mask, output: shadow free image). I successfully created the face mask with the Bisenet and removed the background from the portrait image. In the next step I received the latent vectors from StyleGAN.

My question now is: How do you explore the latent space to find the relevant parts of the vector which control the shadows? You create K random latent vectors but what is your strategy? How many values do you manipulate in every sample? Any hint would be very helpful to me! Thanks in advance.

[YingqingHe]

Hi.
(1) Our method does not need to find the shadow part in the latent space. Instead, we aim to find the latent vector which represent the clean face with no shadow. In order to do this, we propose a 3-stage optimization process to get the latent vector only for the shadow-free face (which details can be found in our paper).
(2) In stage1, we create 500 latent vectors only for a better initialization of the latent vector, and we choose a best one which generated image has the lowest perceptural loss with the input image.
(3) Actually our method is not based on manipulation in latent space. We explicitly model the shadow generation process via the color matrix and the shadow mask. More details can be checked in our paper. Thanks!

[SleyAI]

Hello, thank you very much for your reply! Can you explain how you receive the Initial shadow free image after step 1? In your optimization process in step 1 you calculate the LPIPS loss between the generated images and the original image (which contains shadows). How can you generate a shadow free image by comparing with a shadowed image?

[YingqingHe]

Hi， Just by early stopping. In step 1 we only want to rough version of face, so just stop optimization of latent code after a few of steps. We can obtain a more precise clean face(step 3) after the shadow has been learned(step 2). Best, Yingqing Henning ***@***.***>于2022年1月19日 周三21:55写道：

…

Hello, thank you very much for your reply! Can you explain how you receive the Initial shadow free image after step 1? In your optimization process in step 1 you calculate the LPIPS loss between the generated images and the original image (which contains shadows). How can you generate a shadow free image by comparing with a shadowed image? — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATFIFPONID5TQI7HSSWXRLTUW27EDANCNFSM5KOIPYNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: <YingqingHe/Shadow-Removal-via-Generative-Priors/issues/1/1016488365@ github.com>