README.md

The effect of shape and illumination on material perception: model and applications

Code for Siggraph 2021 paper: The effect of shape and illumination on material perception: model and applications.

The effect of shape and illumination on material perception: model and applications

Abstract

Material appearance hinges on material reflectance properties but also surface geometry and illumination. The unlimited number of potential combinations between these factors makes understanding and predicting material appearance a very challenging task. In this work, we collect a large-scale dataset of perceptual ratings of appearance attributes with more than 215,680 responses for 42,120 distinct combinations of material, shape, and illumination. The goal of this dataset is twofold. First, we analyze for the first time the effects of illumination and geometry in material perception across such a large collection of varied appearances. We connect our findings to those of the literature, discussing how previous knowledge generalizes across very diverse materials, shapes, and illuminations. Second, we use the collected dataset to train a deep learning architecture for predicting perceptual attributes that correlate with human judgments. We demonstrate the consistent and robust behavior of our predictor in various challenging scenarios, which, for the first time, enables estimating perceived material attributes from general 2D images. Since the predictor relies on the final appearance in an image, it can compare appearance properties across different geometries and illumination conditions. Finally, we demonstrate several applications that use our predictor, including appearance reproduction using 3D printing, BRDF editing by integrating our predictor in a differentiable renderer, illumination design, or material recommendations for scene design.

Environment Setting it up

conda env create -f environment.yml -p 'your_env_path'

Dependencies

First, clone and install dependencies

# clone project   
git clone https://github.com/Hans1984/material-illumination-geometry.git      

Get Dataset

• download dataset dataset

How to test

Here we give several test images with labels under the folder ./test.
Run:

python main.py

How to Train a new model

Edit the config.py, put training data under 'input_path' and labels under gt_path.
Run:

python main.py --is_test False

Predictor Examples

Here are testing examples, we show the predictor results with human annotation.

[update 30/11/2021]

We are introducing some minor fixes to the code. These changes only produce slight variations of the predictions reported in the paper and should not affect the overall quality of the predictions:
Code: We have updated the resizing function from scipy to opencv to avoid reported issues with scipy/scipy#4458.
Code: We have updated the code for selecting the test set so that it matches what we report in the paper (Sec. 5.2).
Pre-trained model: We have updated the pre-trained model with these new changes applied.
Results: The updated prediction errors with these changes (Table 2 in the main paper) are as follows.

	Glossiness	Contrast of reflections	Sharpness of reflections	Metallicness	Lightness	All
A	0.1279 ± 0.1414	0.0957 ± 0.1306	0.1042 ± 0.1425	0.0821 ± 0.1241	0.1373 ± 0.1164	0.1149 ± 0.0958
B	0.3308 ± 0.2346	0.2878 ± 0.2027	0.2358 ± 0.1629	0.2532 ± 0.1987	0.2179 ± 0.1702	0.2540 ± 0.1323

Citation

If you found this code useful please cite our work as:

@article{SIGG2021_Materials,
        author = {Serrano, Ana and Chen, Bin and Wang, Chao, and Piovarci, Michal and Seidel, Hans-Peter and Didyk, Piotr and Myszkowski, Karol},
        title = {The effect of shape and illumination on material perception: model and applications},
        journal = {ACM Trans. on Graph.},
        year = {2021},
    }