Synthetic multi view facial generator

docs/reference/index.md

@@ -67,11 +67,15 @@ Neither the copyright holder nor any applicable licensor will be liable for any
         - [single_demo_grasp Module](single-demonstration-grasping.md)        
 
     - `simulation` Module
+        - [human_model_generation Module](human_model_generation.md)
+    - `data_generation` Module
+        - [synthetic_facial_image_generation Module](synthetic_facial_image_generator.md)
         - [human_model_generation Module](human-model-generation.md)
     - `utils` Module
         - [Hyperparameter Tuning Module](hyperparameter_tuner.md)
-- `Stand-alone Utility Frameworks`
-    - [Engine Agnostic Gym Environment with Reactive extension (EAGERx)](eagerx.md)
+    - `Stand-alone Utility Frameworks`
+        - [Engine Agnostic Gym Environment with Reactive extension (EAGERx)](eagerx.md)
+
 - [ROSBridge Package](rosbridge.md)
 - [C Inference API](c-api.md)
     - [data.h](c-data-h.md)

docs/reference/synthetic_facial_image_generator.md

@@ -0,0 +1,61 @@
+## synthetic_facial_image_generator module
+
+The *synthetic_facial_image_generator* module contains the *MultiviewDataGeneration* class, which implements the multi-view facial image rendering operation.
+
+### Class MultiviewDataGeneration
+
+The *MultiviewDataGeneration* class is a wrapper of the Rotate-and-Render [[1]](#R-R-paper) photorealistic multi-view facial image generator based on the original
+[Rotate-and-Render implementation](https://github.com/Hangz-nju-cuhk/Rotate-and-Render).
+It can be used to perform multi-view facial image generation from a single view image on the wild (eval). 
+The [MultiviewDataGeneration](#projects.data_generation.synthetic-multi-view-facial-image-generation.3ddfa.SyntheticDataGeneration.py ) class has the
+following public methods:
+
+#### `MultiviewDataGeneration` constructor
+```python
+MultiviewDataGeneration(self, args)
+```
+
+Constructor main parameters *args* explanation:
+
+- **path_in**: *str, default='./example/Images'* \
+An absolute path (path in) which indicates the folder that contains the set of single view facial image snapshots to be processed by the algorithm.
+- **path_3ddfa**: *str, default='./'* \
+An absolute path (path 3ddfa) which indicates the 3ddfa module folder of the software structure as presented in the repository. This path is necessary in order 
+for the software to create the folders for the intermediate / temporary storage of files generated during the pre-processing such as 3d face models, facial landmarks etc 
+in the folder results of this path.
+- **save_path**: *str, default='./results'* \
+The output images are stored in the folder indicated by save path which is also a class input parameter.
+- **val_yaw**: *str, default='10,20'* \
+Definition of the yaw angles (in the interval [−90◦,90◦]) for which the rendered images will be produced.
+- **val_pitch**: *str, default=' 30,40'* \
+Definition of the pitch angles (in the interval [−90◦,90◦]) for which the rendered images will be produced.
+- **device**: *{'cuda', 'cpu'}, default='cpu'* \
+Specifies the device to be used.
+
+
+#### `MultiviewDataGeneration.eval`
+```python
+MultiviewDataGeneration.eval()
+```
+
+This function is implementing the main procedure for the creation of the multi-view
+facial images, which consists of three different stages. Instead of initializing the main
+parameters of the 3DDFA network in the intializer , the first stage includes detection of the
+candidate faces in the input images and 3D-head mesh fitting using 3DDFA. Moreover,
+the second stage extracts the facial landmarks in order to derive the head pose and align
+the images with the 3d head model mesh. Finally, the main functionality of the multiview
+facial image rendering is executed by loading the respective network parameters.
+
+### Usage Example
+
+```python
+python3 tool_synthetic_facial_generation.py -path_in ./demos/imgs_input/ -path_3ddfa ./algorithm/DDFA/ -save_path ./results -val_yaw 10, 40 -val_pitch 10, 30 -device cuda
+```
+The corresponding paths for the input, output folders as well as the pitch and yaw angles for which the user wants to
+produce the facial images can be easily incorporated in the class creation while the method is initialized. 
+The process is executed for the CNN parameters and GPUs specified in the arguments of the aforementioned command. Users that wish to modify these parameters shall change the respective input arguments which derived from a parser including the arguments path in, path_3ddfa, save_path, val_yaw, val_pitch etc. 
+
+#### References
+<a name="R-R-paper" href="https://github.com/Hangz-nju-cuhk/Rotate-and-Render">[1]</a>
+Hang Zhou, Jihao Liu, Ziwei Liu, Yu Liu, Xiaogang Wang, Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images,
+[arXiv](https://arxiv.org/abs/2003.08124#).  

projects/data_generation/synthetic_multi_view_facial_image_generation/README.md

@@ -0,0 +1,71 @@
+# Synthentic Multi-view Facial Image Generation based on Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images (CVPR 2020)
+
+Based on: [[Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images]](https://arxiv.org/abs/2003.08124)
+
+We utilize, with small modifications in order to be easily executed, publicly available code, namely an un-supervised framework that can synthesize photorealistic rotated facial images using as input  a single facial image, or multiple such images (one per person).
+The implemented method allows for rotating faces in the 3D space back and forth, and then re-rendering them to the 2D plane.
+The generated multi-view facial images can be used for different learning tasks, such as in self-supervised learning tasks.
+
+## Sources:
+* Face Alignment in Full Pose Range: A 3D Total Solution (IEEE TPAMI 2017)
+* Neural 3D Mesh Renderer (CVPR 2018)
+* Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images (CVPR 2020)
+
+## Requirements
+* Python 3.6 is used. Basic requirements are listed in the 'requirements.txt'.
+
+```
+pip3 install -r requirements.txt
+```
+* Install the [Neural_Renderer](https://github.com/daniilidis-group/neural_renderer) following the instructions.
+```
+pip install git+https://github.com/cidl-auth/neural_renderer
+```
+
+* Download checkpoint and BFM model from [checkpoint.zip](ftp://opendrdata.csd.auth.gr/data_generation/synthetic_multi-view-facial-generator/ckpt_and_bfm.zip) put it in ```3ddfa``` and unzip it:
+```bash
+wget ftp://opendrdata.csd.auth.gr/data_generation/synthetic_multi-view-facial-generator/checkpoints.zip
+unzip checkpoints.zip
+unzip checkpoints/ckpt_and_bfm.zip -d 3ddfa
+```
+The 3D models are borrowed from [3DDFA](https://github.com/cleardusk/3DDFA).
+
+* Compile cython code and download remaining models:
+```bash
+cd algorithm/DDFA/utils/cython/
+python3 setup.py build_ext -i
+cd ../../../..
+mkdir algorithm/DDFA/models
+mkdir algorithm/DDFA/example
+wget https://github.com/cleardusk/3DDFA/blob/master/models/phase1_wpdc_vdc.pth.tar?raw=true -O algorithm/DDFA/models/phase1_wpdc_vdc.pth.tar
+```
+
+## Usage Example
+
+1.	Execute the one-step OPENDR function ```tool_synthetic_facial_generation.py``` specifying the input images folder, the output folder, the desired degrees (range -90 to 90) for generating the facial images in multiple view angles pitch and yaw as indicated in the command line: 
+```sh
+python3 tool_synthetic_facial_generation.py -path_in ./demos/imgs_input/ -path_3ddfa ./algorithm/DDFA/ -save_path ./results -val_yaw 10, 40 -val_pitch 10, 30 -device cuda
+```
+
+3. The results can be found in ```results/rs_model/example/```, where multi-view facial images are generated for every person in a respective folder.
+
+
+## License 
+The usage of this software code Rotate-and-Render is under [CC-BY-4.0](https://github.com/Hangz-nju-cuhk/Rotate-and-Render/blob/master/LICENSE).
+The usage of this software code SPADE, SyncBN, 3DDFA are under [MIT License](https://github.com/tasostefas/opendr_internal/blob/synthetic-multi-view-facial-generator/projects/data_generation/synthetic-multi-view-facial-image-generation/3ddfa/LICENSE)
+
+## Acknowledgement
+Large parts of the code are taken from: 
+* The structure of this codebase is borrowed from [SPADE](https://github.com/NVlabs/SPADE).
+* The [SyncBN](https://github.com/vacancy/Synchronized-BatchNorm-PyTorch) module is used in the current code.
+* The [3DDFA](https://github.com/cleardusk/3DDFA) implementation for 3D reconstruction.
+* The code [Rotate-and-Render](https://github.com/Hangz-nju-cuhk/Rotate-and-Render/)  
+
+with the following modifications to make them compatible with the OpenDR specifications:
+## Minor Modifications
+1. All scripts: PEP8 changes
+2. ```3ddfa/preprocessing_1.py, 3ddfa/preprocessing_2.py, test_multipose.py``` Modified to work as a callable functions
+3. ```options/base_options.py, options/test_options.py ``` Commented out/change several parameters to be easily executed 
+4. ```models/networks/render.py``` Minor functional changes
+5. The OPENDR created functions are ```SyntheticDataGeneration.py, tool_synthetic_facial_generation.py```
+6. The rest are taken from the aforementioned repositories

projects/data_generation/synthetic_multi_view_facial_image_generation/SyntheticDataGeneration.py

@@ -0,0 +1,141 @@
+# Copyright 2020-2021 OpenDR European Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# MIT License
+#
+# Copyright (c) 2019 Jian Zhao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+# !/usr/bin/env python3.7
+# coding: utf-8
+from tqdm import tqdm
+from shutil import copyfile
+import cv2
+import os
+from algorithm.DDFA import preprocessing_1
+from algorithm.DDFA import preprocessing_2
+from algorithm.Rotate_and_Render import test_multipose
+
+
+class MultiviewDataGeneration():
+
+    def __init__(self, args):
+
+        self.path_in = args.path_in
+        self.key = str(args.path_3ddfa + "/example/Images/")
+        self.key1 = str(args.path_3ddfa + "/example/")
+        self.key2 = str(args.path_3ddfa + "/results/")
+        self.save_path = args.save_path
+        self.val_yaw = args.val_yaw
+        self.val_pitch = args.val_pitch
+        self.args = args
+
+    def eval(self):
+
+        # STAGE No1 : detect faces and fitting to 3d mesh by main.py execution
+        list_im = []
+
+        print("START")
+
+        a = open("file_list.txt", "w")
+        for subdir, dirs, files in os.walk(self.path_in):
+            current_directory_path = os.path.abspath(subdir)
+            for file in files:
+                name, ext = os.path.splitext(file)
+                if ext == ".jpg":
+                    current_image_path = os.path.join(current_directory_path, file)
+                    current_image = cv2.imread(current_image_path)
+                    list_im.append(current_image_path)
+                    a.write(str(file) + os.linesep)
+                    cv2.imwrite(os.path.join(self.key, file), current_image)
+            self.args.files = list_im.copy()
+            list_im.clear()
+            preprocessing_1.main(self.args)
+        a.close()
+
+        # STAGE No2: Landmarks Output with inference.py execution
+
+        im_list2 = []
+        d = open(os.path.join(self.key1, 'realign_lmk'), "w")
+        for subdir, dirs, files in os.walk(self.path_in):
+            current_directory_path = os.path.abspath(subdir)
+            self.args.img_prefix = current_directory_path
+            self.args.save_dir = os.path.abspath(self.key2)
+            self.args.save_lmk_dir = os.path.abspath(self.key1)
+            if not os.path.exists(self.args.save_dir):
+                os.mkdir(self.args.save_dir)
+            if not os.path.exists(self.args.save_lmk_dir):
+                os.mkdir(self.args.save_lmk_dir)
+
+            list_lfw_batch = './file_list.txt'
+            dst = os.path.join(self.args.save_lmk_dir, "file_list.txt")
+            copyfile(list_lfw_batch, dst)
+            b = open("txt_name_batch.txt", "w")
+            for file in files:
+
+                with open(list_lfw_batch) as f:
+                    img_list = [x.strip() for x in f.readlines()]
+
+                    for img_idx, img_fp in enumerate(tqdm(img_list)):
+                        if img_fp == str(file):
+                            im_list2.append(str(file))
+                            b.write(str(file) + os.linesep)
+            self.args.img_list = './txt_name_batch.txt'
+            b.close()
+            self.args.dump_lmk = 'true'
+            im_list2.clear()
+            preprocessing_2.main(self.args)
+            with open(os.path.join(self.args.save_lmk_dir, 'realign_lmk_')) as f:
+                img_list = [x.strip() for x in f.readlines()]
+                for img_idx, img_fp in enumerate(tqdm(img_list)):
+                    d.write(img_fp + os.linesep)
+        d.close()
+
+        # STAGE No3: Generate Facial Images in specific pitch and yaw angles
+        test_multipose.main(self.save_path, self.val_yaw, self.val_pitch)
+
+    def fit(self):
+        raise NotImplementedError()
+
+    def infer(self):
+        raise NotImplementedError()
+
+    def load(self):
+        raise NotImplementedError()
+
+    def optimize(self):
+        raise NotImplementedError()
+
+    def reset(self):
+        raise NotImplementedError()
+
+    def save(self):
+        raise NotImplementedError()

projects/data_generation/synthetic_multi_view_facial_image_generation/algorithm/DDFA/BFM_Remove_Neck/bfm_show.m

@@ -0,0 +1,14 @@
+model = load('model_refine.mat');
+model = model.model_refine;
+
+mu = model.mu_shape + model.mu_exp;
+mu = reshape(mu, 3, length(mu) / 3);
+tri = model.tri;
+keypoints = model.keypoints;
+pts68_3d = mu(:, keypoints);
+
+render_face_mesh(mu, tri, pts68_3d);
+
+A = getframe(gcf);
+mimg = A.cdata;
+imwrite(mimg, 'imgs/bfm_noneck.jpg', 'quality', 95);

projects/data_generation/synthetic_multi_view_facial_image_generation/algorithm/DDFA/BFM_Remove_Neck/readme.md

@@ -0,0 +1,24 @@
+**Todo:** Update the neck-removing processing pipeline from original BFM model.
+
+The original version with neck:
+<p align="center">
+  <img src="imgs/bfm_noneck.jpg" alt="neck" width="400px">
+</p>
+
+[bfm.ply](https://github.com/Hangz-nju-cuhk/Rotate-and-Render/blob/master/3ddfa/BFM_Remove_Neck/bfm.ply)
+
+The refined version without neck:
+<p align="center">
+  <img src="imgs/bfm_refine.jpg" alt="no neck" width="400px">
+</p>
+
+[bfm_refine.ply](https://github.com/Hangz-nju-cuhk/Rotate-and-Render/blob/master/3ddfa/BFM_Remove_Neck/bfm_refine.ply)
+These two images are rendered by MeshLab.
+
+`bfm_show.m` shows how to render it with 68 keypoints in Matlab.
+
+<p align="center">
+  <img src="imgs/bfm_refine.jpg" alt="no neck">
+</p>
+
+Attention: the z-axis value of `bfm.ply` and `bfm_refine.ply` file are opposed in `model_refine.mat`, do not use these two `ply` file in training.

projects/data_generation/synthetic_multi_view_facial_image_generation/algorithm/DDFA/BFM_Remove_Neck/render_face_mesh.m

@@ -0,0 +1,22 @@
+function render_face_mesh(vertex, tri, pts68_3d)
+    trisurf(tri', vertex(1, :), vertex(2, :), vertex(3, :), ones(size(vertex, 2),1), 'edgecolor', 'none');
+
+    re=[1 1 1];
+    colormap(re);
+
+    light('Position', [0 0 1], 'Style', 'infinite');
+    lighting gouraud
+    axis equal
+    view([0 90]);
+
+    if nargin == 3
+        hold on; plot3(pts68_3d(1,:), pts68_3d(2,:), pts68_3d(3,:)+1, '*');
+    end
+
+    xlabel('x');
+    ylabel('y');
+    zlabel('z');
+
+    axis on;
+    grid on;
+end

projects/data_generation/synthetic_multi_view_facial_image_generation/algorithm/DDFA/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 Jianzhu Guo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.