Skip to content
API to run VirtualHome, a simulator to generate videos of human activities
Branch: master
Clone or download
Latest commit 8853567 May 21, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
assets updates readme May 3, 2019
dataset updates readme May 3, 2019
dataset_augmentation updates clean code Mar 25, 2019
demo Delete unity_demo.py May 21, 2019
docker updates readme Mar 25, 2019
example_graphs added trimmed scenes Mar 15, 2019
resources corrected examples Mar 25, 2019
simulation more simplified code May 21, 2019
.gitignore add .DS_Store Mar 13, 2019
README.md Update README.md May 21, 2019
requirements.txt Update requirements.txt May 3, 2019

README.md

VirtualHome

VirtualHome is a platform to simulate complex household activities via programs. Given an initial environment describing an apartment and a program depicting a sequence of actions, VirtualHome executes the program generating a video of the activity together with useful data for activity understanding or planning. Check out more details of the environmnent and platform in www.virtual-home.org.

Cite VirtualHome

VirtualHome has been used in:

  • VirtualHome: Simulating HouseHold Activities via Programs, CVPR2018
  • Synthesizing Environment-Aware Activities via Activity Sketches, CVPR2019

If you plan to use the simulator, please consider citing them.

@inproceedings{puig2018virtualhome,
  title={Virtualhome: Simulating household activities via programs},
  author={Puig, Xavier and Ra, Kevin and Boben, Marko and Li, Jiaman and Wang, Tingwu and Fidler, Sanja and Torralba, Antonio},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={8494--8502},
  year={2018}
}
@inproceedings{puig2018virtualhome,
  title={Virtualhome: Simulating household activities via programs},
  author={Puig, Xavier and Ra, Kevin and Boben, Marko and Li, Jiaman and Wang, Tingwu and Fidler, Sanja and Torralba, Antonio},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={8494--8502},
  year={2018}
}

Contents

  • Overview
  • Set Up
  • Generating Videos/Keyframes
  • Dataset
  • Script Augmentation
  • Other details

Overview

Activities in VirtualHome are represented through two components: programs representing the sequence of actions that compose an activity, and graphs representing a definition of the environment where the activity takes place. Given a program and a graph, the simulator executes the program, generating a video of the activity or a sequence of graphs representing how the environment evolves as the activity takes place. To this end, VirtualHome includes two simulators: the Unity Simulator and Evolving Graph.

Unity Simulator

This simulator is built in Unity and allows generating videos of activities. To use this simulator, you will need to download the appropiate executable and run it with the Python API. You can check a demo of the simulator in demo/unity_demo.ipynb

Evolving Graph

This simulator runs fully in python and allows to generate a sequence of graphs when a program is executed. You can run it in simulation/evolving_graph. Note that some of the objects and actions in this simulator are not supported yet in Unity Simulator.

Set Up

Clone repository and install dependencies

pip install -r requirements.txt
git clone https://github.com/xavierpuigf/virtualhome.git

Download Unity Simulator

Download the VirtualHome UnitySimulator executable and move it under simulation/unity_simulator.

Test simulator

Double click the executable and select a resolution and screen size. Then, run the demo in demo/unity_demo.ipynb. If you do not have a Monitor, you can either use Docker or use an X server. Following this medium post. When running the executable, use -batchmode.

You can also test the Evolving Graph simulator in demo/example.py. This simulator does not require opening any executable if you do not plan to generate images.

Docker

You can also run Unity Simulator using Docker. You can find how to set it up here.

Generating Videos and Snapshots

VirtualHome Unity Simulator allows generating videos corresponding to household activities. In addition, it is possible to use Evolving Graph simulator to obtain the environment for each execution step and use UnitySimulator to generate snapshots of the environment at each step.

Generate videos

Open the simulator and run:

cd demo/
python generate_video.py

Generate snapshots

Open the simulator and run:

cd demo/
python generate_snapshots.py

A grid of snapshots for the given script will be generated.

Dataset

We collected a dataset of programs and augmented them with graphs using the Evolving Graph simulator. You can download them here. Once downloaded, move the programs into the dataset folder. The dataset should follow the following structure:

dataset
└── programs_processed_precond_nograb_morepreconds
	|── initstate
	├── withoutconds
	├── executable_programs
	|   ├── TrimmedTestScene7_graph
	|	└── ...
	└── state_list
		├── TrimmedTestScene7_graph
	   	└── ...	

The folders withoutconds and initstate contain the original programs and pre-conditions.

When a script is executed in an environment, the script changes by aligning the original objects with instances in the environment. You can view the resulting script in executable_programs/{environment}/{script_name}.txt.

To view the graph of the environment, and how it changes throughout the script execution of a program, check state_list/{environment}/{script_name}.json.

You can find more details of the programs and environment graphs in dataset/README.md.

Script Augmentation

In Synthesizing Environment-Aware Activities via Activity Sketches, we augment the scripts with two knowledge bases: KB-RealEnv and KB-ExceptonHandler. You can download the augmented scripts in KB-RealEnv and KB-ExceptionHandler.

Here, we provide the code to augment the sripts:

Augment with KB-RealEnv

cd dataset_generation
python augment_dataset_locations.py

Augment with KB-ExceptionHandler

cd dataset_generation
python augment_dataset_exceptions.py

Resources

To do the above generation and augmentation, some valuable resource files are used to set the properties of objects, set the affordance of objects, etc. Check resources/README.md for more details.

You can’t perform that action at this time.