GIIN: Graph-based Image Inpainting
The project goal was to explore the applications of spectral graph theory to address the inpainting problem of large missing chunks. We used a non-local patch graph representation of the image and proposed a structure detector which leverages the graph representation and influences the fill-order of our exemplar-based algorithm. Our method achieved state-of-the-art performances.
- Identify the potentially fillable unknown pixels.
- Give a priority to each of them and identify the one with the highest priority. We will inpaint that pixel.
- Find in the image and copy the patch which is the most similar to the patch around the chosen pixel.
Our innovation resides in the way to assign a priority to the unknown pixels. We want to assign a higher priority to the structure than to the texture in order to preserve it while reconstructing. We do it the following way:
- Construct a graph of pixels where the edge weights are defined as a similarity measure between the connected pixels.
- Place a dirac on the considered pixel.
- Diffuse the dirac with a heat kernel. As seen on the image, the diffused signal is constrained by the cluster around the tested pixel. From there, we should be able to determine if the considered pixel is part of a texture or a structure element. Looking at the spreading of the diffusion, texture and structure pixels are easily tell apart.
- The priority is then given by the l2 norm of the diffused signal. See the report for the motivation and mathematical definition.
- We finally add a confidence priority so that pixels who are farther away from the known pixels get lower priorities.
- GSPbox: graph signal processing toolbox (version 0.4.0 should work)
- UNLocBoX: convex optimization toolbox
The FLANN library implements fast algorithms to approximate nearest neighbors
search. It can be used to speed up the graph construction. It is optional and
if you do not want to use it you can comment
param.nnparam.use_flann = 1; in
lib/giin_patch_graph.m. Otherwise compile it this way:
cd gspbox/3rdparty/sources/flann-1.8.4-src/ mkdir build && cd build cmake .. make
Make sure that the MATLAB bin folder is in your path to compile the MEX. If it
did work you should have a file called
Place a file
datasub-folder. The masked area should be bright green, i.e. RGB [0,255,0]. You may want to generate some synthetic images with e.g.
Launch the inpainting process from MATLAB:
Or from the shell:
./launch.sh inpaint vertical
You may then want to adjust the paths to the toolboxes in