Joint Embeddings of Shapes and Images via CNN Image Purification
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

Joint Embeddings of Shapes and Images via CNN Image Purification

Created by Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Leonidas J. Guibas from Stanford University, and Noa Fish, Daniel Cohen-Or from Tel Aviv University.


We propose a way to embed 3D shapes and 2D images into a joint embedding space, thus all of the 3D shapes and 2D images become searchable from each other (live demo). The research paper was accepted to SIGGRAPH Asia 2015.


JointEmbedding is released under the 4-clause BSD license (the original "BSD License", refer to the LICENSE file for details).

Citing JointEmbedding

If you find JointEmbedding useful in your research, please consider citing:

    Author = {Li, Yangyan and Su, Hao and Qi, Charles Ruizhongtai and Fish, Noa
        and Cohen-Or, Daniel and Guibas, Leonidas J.},
    Title = {Joint Embeddings of Shapes and Images via CNN Image Purification},
    Journal = {ACM Trans. Graph.},
    Year = {2015}


1. Usage: How to test with trained models?

The trained model for chair category (03001627) can be downloaded from here. You can try them with script and in src/image_embedding_testing folder, with the following commands: --image YOUR_TESTING_IMAGE --caffemodel image_embedding_03001627.caffemodel  --prototxt image_embedding_03001627.prototxt --image YOUR_TESTING_IMAGE --caffemodel image_embedding_03001627.caffemodel  --prototxt image_embedding_03001627.prototxt

2. Usage: How to train your own models?

2.1. Requirements: datasets

  • ShapeNetCore is used for constructing the shape embedding space and generating synthetic images. Visit The, and request to download the ShapeNetCore dataset. ShapeNetCore.v1 (also called ShapeNetCore2015Summer) is prefered (there were many broken meshes in ShapeNetCore.v0/ShapeNetCore2015Spring).
  • SUN2012 dataset is used for background overlay of the synthetic images. Will be downloaded by our script.

2.2. Requirements: software

  • Caffe is used for deep learning. Install it (including the pycaffe module) by following their instructions. You are required to specify your caffe installation path in
  • Blender is used for rendering shapes into images. Will be downloaded by our script.
  • Matlab. You are required to specify matlab executable path in
  • Piotr's Image & Video Matlab Toolbox is used for HoG feature extraction. Will be downloaded by our script.

2.3. Requirements: hardware

  • Highend GPU(s) are required for the deep learning part.
  • You may also need highend CPU(s), as millions of images will be rendered, processed.

2.4. Installation

The code is written by python, matlab and shell. There is no need for any installation of the code itself. Just:

git clone JointEmbedding;
cd JointEmbedding/src;

2.5. Run the pipeline

  1. Edit, especially the ones marked by [take care!]
  2. Execute It will download some 3rd party software, and prepare shell scripts for next steps.
  3. Execute to generate shape embedding space.
  4. Execute to generate synthetic images.
  5. Execute to prepare and start the actual process.
  1. You can run step 3 and 4 in parallel, well, in different machines, since both of them are multi-threaded, and you won't gain much speedup if you run them in parallel in the same machine.
  2. Step 3, 4, and 5 are also very I/O intensive, try large SSD if you have.
  3. The run_\*.sh scripts further divided the tasks into smaller tasks. Feed -f first_step -l last_step parameters to the run_\*.sh scripts to run part of them.
  4. Read the scripts, starting from the run_\*.sh, to get more understanding of the code and build upon it!

3. Questions?

Refer to Frequently Asked Questions first.