UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion(CVPR 2024)

Junsheng Zhou* · Weiqi Zhang* · Baorui Ma · Kanle Shi · Yu-Shen Liu · Zhizhong Han

(* Equal Contribution)

Paper | Project Page

We will release the code of the paper UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion in this repository.

Abstract

In this work, we present UDiFF, a 3D diffusion model for unsigned distance fields (UDFs) which is capable to generate textured 3D shapes with open surfaces from text conditions or unconditionally. Our key idea is to generate UDFs in spatial-frequency domain with an optimal wavelet transformation, which produces a compact representation space for UDF generation. Specifically, instead of selecting an appropriate wavelet transformation which requires expensive manual efforts and still leads to large information loss, we propose a data-driven approach to learn the optimal wavelet transformation for UDFs.

Method

Overview of UDiFF. (a) We propose a data-driven approach to attain the optimal wavelet transformation for UDF generation. We optimize wavelet filter parameters through the decomposition and inversion by minimizing errors in UDF self-reconstruction. (b) We fix the learned decomposition wavelet parameters and leverage it to prepare the data as a compact representation of UDFs including pairs of coarse and fine coefficient volumes. (c) is the architecture of the generator in diffusion models, where text conditions are introduced with cross-attentions. (d) The diffusion process of UDiFF. We train the generator to produce coarse coefficient volumes from random noises guided by input texts and train the fine predictor to predict fine coefficient volumes from the coarse ones.

Visualization Results

Category Conditional Generations

Outfit Designs with UDiFF Garment Generations

Generation Results

DeepFashion3D

​ ​

Category conditional generations.

ShapeNet dataset

Unconditional generations.

Comparison Results

Citation

If you find our code or paper useful, please consider citing

@inproceedings{udiff,
    title={UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusion},
    author={Zhou, Junsheng and Zhang, Weiqi and Ma, Baorui and Shi, Kanle and Liu, Yu-Shen and Han, Zhizhong},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2024}
}