Neural Scene De-rendering
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

Neural Scene De-rendering

This repository contains scripts for rendering Minecraft images as those in the CVPR'17 Neural Scene De-rendering paper.


Project Malmo: Please follow the instructions from Malmo to launch the Minecraft client (0.30.0).

Please also include the path to Malmo schemas in the environment variable MALMO_XSD_PATH.

Torch: We use Torch 7 for our implementation with these additional packages:

  • libMalmoLua: offered in Project Malmo. Its path should be included in the environment variable LUA_CPATH.


Our current release has been tested on Ubuntu 14.04.

Cloning the repository

git clone
cd nsd

Connecting to the client

Please update line 101 of mc-gen.lua with the actual IP and port of your Minecraft client.

Image rendering demo (main.lua)

The demo calls the rendering function to render an image based on the scene XML. Please find descriptions of the scene XML we used in the source file.

th main.lua 

The script renders the image and saves it as demo.png. It should look as follows.

Sometimes the image might look darker than expected, in which cases please try again.


  title={{Neural Scene De-rendering}},
  author={Wu, Jiajun and Tenenbaum, Joshua B and Kohli, Pushmeet},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},

For any questions, please contact Jiajun Wu (