VirtualHome2KG is a system for constructing and augmenting knowledge graphs (KGs) of daily living activities using virtual space. We also provide an ontology to describe the structure of the KGs. We used VirtualHome as a platform of virtual space simulation. Thus, this repository is an extension of the virtualhome. Please see the original repository of the virtualhome for details of the Unity simulation.
- Overview
- Set Up
- How to use
- Schema
- Beta version: new schema (Event-centric. more information)
- stable version: old schema (Activity-centric. presented at ISWC2021 Poster, ICTAI2021 Short paper)
git clone https://github.com/aistairc/virtualhome2kg.gitWe provide VirtualHome2KG as Jupyter notebooks. If you want to run the tools, install Jupyter and run it on your host.
First of all, you need to build an environment where VirtualHome can run.
pip install -r requirements.txtSecond, you need to download the VirtualHome Unity Simulater and move it under simulation/ (for MacOS) or simulation/unity_simulator (Windows).
Please see the below link.
Download Unity Simulator
Our experimental conditions:
Local machine
- Python: anaconda3-2020.07
- PC: MacBook Pro (13-inch, 2020, Four Thunderbolt 3 ports)
- OS: macOS Catalina (10.15.7)
- CPU: 2.3 GHz Quad core Intel Core i7
- Memory: 32GB 3733 MHz LPDDR4X
- GPU: Intel Iris Plus Graphics 1536 MB
Server
- Triplestore: GraphDB SE 9.6 by Ontotext
- OS: Windows 10 for Workstation
- CPU: Xeon Gold 5215L 10 cores
- Memory: 2.15TB
If you want to generate knowledge graphs based on the VirtualHome's activities, you need to downlowd the dataset.
First, you have to run the Unity simulator: https://github.com/xavierpuigf/virtualhome#download-unity-simulator.
Second, you have to download the dataset of the VirtualHome from here, and move them to dataset/.
We provide two scripts for generating activity graphs (Note that the activity graph is not a knowledge graph. This is the simulation results described in JSON format).
The first one generates the activity graphs without recording objects' 3D coordinates and actions' duration (execution time). This script simulates without rendering on the Unity simulator to generate a lot of data in a short time.
Jupyter notebook:
Run demo/generate_activitiy_graph.ipynb
The second one generates the activity graphs containing 3D coordinates and duration. This version simulates using the Unity simulator. Therefore, it takes a lot of time to generate the data. In addition, the executability of the simulation depends on the VirtualHome's Unity simulator.
Jupyter notebook:
Run demo/generate_activitiy_graphi_with_bbox.ipynb
Jupyter notebook:
Run demo/create_rdf_r3.ipyb
If you want to generate RDF file containing objects' bbox.
Jupyter notebook:
Run demo/create_rdf_r3_bbox.ipyb
Jupyter notebook:
Run demo/add_places.ipynb
A visualization tool is available. https://github.com/aistairc/virtualhome2kg_visualization
We designed the knowledge graph schema for the VirtualHome2KG. Please see here for learing more about our schema.
Egami, S., Ugai, T., Oono, M., Kitamura, K., Fukuda.: Synthesizing Event-centric Knowledge Graphs of Daily Activities using Virtual Space. IEEE Access, Early Access. doi: https://doi.org/10.1109/ACCESS.2023.3253807 (2023)
Egami, S., Nishimura, S., Fukuda, K.: A Framework for Constructing and Augmenting Knowledge Graphs using Virtual Space: Towards Analysis of Daily Activities. Proceedings of the 33rd IEEE International Conference on Tools with Artificial Intelligence. pp.1226-1230 (2021) [IEEE Xplore]
Egami, S., Nishimura, S., Fukuda, K.: VirtualHome2KG: Constructing and Augmenting Knowledge Graphs of Daily Activities Using Virtual Space. Proceedings of the ISWC 2021 Posters, Demos and Industry Tracks: From Novel Ideas to Industrial Practice, co-located with 20th International Semantic Web Conference. CEUR, Vol.2980 (2021) [pdf]