[Japanese]
 |
 |
- Provided_Dataset
- Dataset_Composition
- Explanation of Knowledge Graphs
- How to Use Knowledge Graphs
- Creating Similar Knowledge Graphs
- The provided dataset consists of video data that depicts various daily life actions in VirtualHome.
- The data is transformed into knowledge graphs that provide information about "who" performed the "action", "what" was involved, and the resulting "state" or "position" of the object.
- This data is available to the public as open data.
- This data set consists of complete data and partially missing data.
- Complete data is the original data built from Script Data.
- PartiallyMissingData is some parcentage(20%, 50%, 100%) of data(place, action and object) anonymized from the original data.
- The directory name 101010 indicates that the location is 100%, the action is 100%, and the object is 100% anonymized.
- The directory name 550 indicates that the location is 50%, the action is 50%, and the object is 0% anonymized.
- The directory name 200 indicates that the location is 20%, the action is 0%, and the object is 0% anonymized.
-
- The videos are in mp4 format
- The videos consist of 709 action scenarios
- For each scenario, there are three types of videos: a character rear view (file name ending in 0), an indoor camera switching view (file name ending in 1), and a fixed camera view placed in each corner of the room (file name ending in 2-5).
- For each action scenario, we generated data for a minimum of 1 to a maximum of 7 patterns with different room layouts (scenes).
- In total, there are 1,224 videos.
- Videos with slowly moving characters simulate the movements of elderly people.
-
- The scene graphs are in Action Genome format.
-
- The knowledge graphs are in RDF format.
- The knowledge graphs consist of 709 knowledge graphs corresponding to videos, along with their their schema and location supplement information
- The schema is described below
- SPARQL endpoints and query examples are available
-
- The script data is in txt format.
- The data is provided to VirtualHome2KG to generate videos and knowledge graphs.
- A script includes the action title and a brief description in text format.
Please refer to the following specification document for detailed description of all classes, instances, and properties: https://aistairc.github.io/VirtualHome2KG/vh2kg_ontology.html
Here are some explanations of typical classes and properties. We provide the descriptions of different categories and features.
Prefixes
Representative Classes
| QName | Description |
| ho:Activity | Human daily activities at home. This class is reused from the HomeOntology. |
| :Event | Smaller events that make up activities. |
| :Action | Human actions (or activities) performed within events. |
| :Object | Various objects found in the home, such as food, furniture, appliances, consumables, and household items. |
| :Situation | The state of the home at a specific moment. Instances of the state (State) of all objects in the home at that moment become part of an instance of this class (part). |
| :State | The state of a specific object at a specific moment. A new instance is only created when the state changes before and after the event. |
| :StateType | The type of state of an object. Instances of this class are based on the object states of VirtualHome. |
| :Attribute | Attributes of an object. |
| :Shape | A class that represents the shape and position of an object, with values for size and coordinates in a 3D bounding box. This class is reused from the X3D ontology. |
| time:Duration | The duration of an action in seconds. This class is reused from the Time Ontology. |
Representative properties
| QName | Domains | Ranges | Description |
| :activty | :Character | :Activity | Associates a character (agent) with an activity. |
| :action | :Event | :Action | Associates an event with an action (motion). Activities are composed of sequences of actions. |
| :eventNumber | :Event | xsd:int | Indicates the order of events in an activity. |
| :situationBeforeEvent | :Event | :Situation | Associates an event with a situation. The household situation before some event is about to occur. |
| :situationAfterEvent | :Event | :Situation | Associates an event with a situation. The household situation after some event has occurred. |
| :mainObject | :Event | :Object | A sub-property of ho:object. Associates an event with its main object. This property is used when specifying the object that is acted upon in an event, for example, "object_X is actioned upon object_Y". |
| :targetObject | :Event | :Object | A sub-property of ho:object. Associates an event with its target object. This property is used when specifying the object that is acted upon in an event, for example, "object_X is actioned upon object_Y". |
| time:hasDuration | :Event | time:Duration | Associates an event with its duration. |
| :isStateOf | :State | :Object | Associates an object with its state. |
| :state | :State | :StateType | Associates a state with its value. |
| :affords | :Object | :Action | Associates an object with an action. Denotes the affordances of an object. |
| :attribute | :Object | :Attribute | Associates an object with an attribute. |
| :partOf | |||
| :partOf | :State | :Situation | Indicates which momentary Situation a State of an object is part of, in the home. |
| :bbox | :State | :Shape | Relates the resource of the shape at a given moment of the State of an object. |
| :nextActivity | :Activity | :Activity | Indicates the relationship between two Activities. The next Activity. |
| :nextEvent | :Event | :Event | Indicates the relationship between two Events. The next Event. |
| :nextSituation | :Situation | :Situation | Indicates the relationship between two Situations. The next Situation. |
| :nextState | :State | :State | Indicates the relationship between two States of an object. The next State of the object. |
| :between | :Shape | :Shape | Primarily applies to door objects. Indicates the relationship between the door and the rooms it separates (e.g., door between kitchen and living room). Refer to VirtualHome's relations for more details. |
| :close | :Shape | :Shape | Indicates that the distance between two objects is within 1.5 meters. Refer to VirtualHome's relations for more details. |
| :facing | :Shape | :Shape | Indicates that an object 2 is visible from object 1. The center of the two objects must be within 5 meters of each other. Refer to VirtualHome's relations for more details. |
| :holds_lh | :Shape | :Shape | Indicates that the object is held in the left hand. The subject is thus a Shape related to a Character. Refer to VirtualHome's relations for more details. |
| :holds_rh | :Shape | :Shape | Indicates that the character is holding an object with their right hand. Therefore, the subject is the shape of the object related to the character. For more details, please refer to the VirtualHome "relations" section in https://github.com/xavierpuigf/virtualhome/tree/master/src/virtualhome/simulation#relations. |
| :inside | :Shape | :Shape | Indicates that object 1 is located inside object 2. For more details, please refer to the VirtualHome "relations" section in https://github.com/xavierpuigf/virtualhome/tree/master/src/virtualhome/simulation#relations. |
| :on | :Shape | :Shape | Indicates that object 1 is located on top of object 2. |
The Knowledge Graph is provided in RDF format and can be stored in a triplestore. The SPARQL query language allows for various searches and a direct endpoint is available.
References:
- Reference material about Triple Stores.: Introduction to Triplestores (page 12)
- Reference materials on SPARQL: Examples of SPARQL Queries using Wikidata (page 16)
An SPARQL endpoint for this dataset is available at https://kgrc4si.home.kg:7200/sparql. Please select "KGRC4SIv05" as the repository. The triplestore used is Ontotext GraphDB.
If this endpoint is down, please use the mirror repository provided by the Kozaki Laboratory at Osaka Electro-Communication University (http://kozaki-lab.osakac.ac.jp/agraph/kgrc4si). Note that it may not have the latest data.
- Retrieve a list of activities
- Retrieve events and actions in the "clean the kitchen" activity
- Frequently grasped objects
- List of types of objects that are interacted with
- Add height information for objects
PREFIX ex: <http://kgrc4si.home.kg/virtualhome2kg/instance/>
PREFIX : <http://kgrc4si.home.kg/virtualhome2kg/ontology/>
select DISTINCT * where {
?activity :virtualHome ex:scene1 .
}PREFIX ex: <http://kgrc4si.home.kg/virtualhome2kg/instance/>
PREFIX : <http://kgrc4si.home.kg/virtualhome2kg/ontology/>
select DISTINCT * where {
ex:clean_kitchen1_scene1 :hasEvent ?event .
?event :action ?action .
}PREFIX : <http://kgrc4si.home.kg/virtualhome2kg/ontology/>
select distinct ?objectType where {
?event (:mainObject|:targetObject) ?object .
?object a ?objectType .
}PREFIX ho: <http://www.owl-ontologies.com/VirtualHome.owl#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX : <http://kgrc4si.home.kg/virtualhome2kg/ontology/>
PREFIX dcterms: <http://purl.org/dc/terms/>
PREFIX ac: <http://kgrc4si.home.kg/virtualhome2kg/ontology/action/>
select ?name (count(?object) AS ?count) where {
?objectClass rdfs:subClassOf/rdfs:subClassOf :Object .
?object a ?objectClass ;
rdfs:label ?label ;
dcterms:identifier ?id .
?event (:mainObject|:targetObject) ?object .
?event :action ac:grab .
BIND(concat(?label, ?id) AS ?name)
} group by ?object ?name order by desc(count(?object))PREFIX x3do: <https://www.web3d.org/specifications/X3dOntology4.0#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX : <http://kgrc4si.home.kg/virtualhome2kg/ontology/>
PREFIX ex: <http://kgrc4si.home.kg/virtualhome2kg/instance/>
CONSTRUCT {
?object :height ?height_node .
?height_node rdf:value ?size_y1 ;
:unit :meter .
} WHERE {
?state1 :isStateOf ?object ; :bbox ?shape1 .
?shape1 x3do:bboxSize ?size1 .
?size1 rdf:rest ?size_y .
?size_y rdf:first ?size_y1 .
BIND(REPLACE(STR(?object), STR(ex:) ,"") AS ?object_name)
BIND(URI(CONCAT(STR(ex:),"height_", ?object_name)) AS ?height_node)
}Note that this affordance information is different from VirtualHome's default settings; please see here if you want to use VirtualHome's affordance information.
# Please turn on inference engine
PREFIX : <http://kgrc4si.home.kg/virtualhome2kg/ontology/>
select * where {
?object a :Object .
?object :affords ?action .
} limit 100 No SPARQL Endpoint for Partially Missing data is provided.
from rdflib import Graph
g = Graph()
file_path = 'https://raw.githubusercontent.com/KnowledgeGraphJapan/KGRC-RDF/kgrc4si/PartiallyMissingData/RDF/101010/Admire_art1_scene1-101010.ttl'
# file_path can be URL or the name of a local file
g.parse(file_path, format='turtle')
# SPARQL Query
sparql_query = """
prefix ex: <http://kgrc4si.home.kg/virtualhome2kg/instance/>
prefix vh2kg: <http://kgrc4si.home.kg/virtualhome2kg/ontology/>
select *
WHERE {
?event a vh2kg:Event
}
"""
# Execute SPARQL with the defined query and retrieve the results
query_result = g.query(sparql_query)
# Display the result
for row in query_result:
print(row)Result
(rdflib.term.URIRef('http://kgrc4si.home.kg/virtualhome2kg/instance/event3_admire_art1_scene1'),)
(rdflib.term.URIRef('http://kgrc4si.home.kg/virtualhome2kg/instance/event1_admire_art1_scene1'),)
(rdflib.term.URIRef('http://kgrc4si.home.kg/virtualhome2kg/instance/event2_admire_art1_scene1'),)
This dataset was created using our proposed system "VirtualHome2KG." For more information, please refer to the document.
- Egami, S., Ugai, T., Oono, M., Kitamura, K., Fukuda.: Synthesizing Event-centric Knowledge Graphs of Daily Activities using Virtual Space. IEEE Access, Vol. 11, pp.23857-23873. doi: https://doi.org/10.1109/ACCESS.2023.3253807 (2023)
- Egami, S., Nishimura, S., Fukuda, K.: A Framework for Constructing and Augmenting Knowledge Graphs using Virtual Space: Towards Analysis of Daily Activities. Proceedings of the 33rd IEEE International Conference on Tools with Artificial Intelligence. pp.1226-1230 (2021) [IEEE Xplore]
- Egami, S., Nishimura, S., Fukuda, K.: VirtualHome2KG: Constructing and Augmenting Knowledge Graphs of Daily Activities Using Virtual Space. Proceedings of the ISWC 2021 Posters, Demos and Industry Tracks: From Novel Ideas to Industrial Practice, co-located with 20th International Semantic Web Conference. CEUR, Vol.2980 (2021) [pdf]
- 江上周作，鵜飼孝典，Swe Nwe Nwe Htun，太田雅輝，大野美喜子，北村光司，松下京群，古崎晃司，川村隆浩，福田賢一郎: 家庭内の日常生活動画とイベント中心知識グラフの同時生成，2023年度人工知能学会全国大会（第37回）, (2023) (In Japanese)
- 江上周作，鵜飼孝典，窪田文也，大野美喜子，北村光司，福田賢一郎: 家庭内の事故予防に向けた合成ナレッジグラフの構築と推論，第56回人工知能学会セマンティックウェブとオントロジー研究会, SIG-SWO-056-14 (2022) [J-STAGE](In Japanese)
Shusaku Egami and others created "VirtualHome2KG dataset - simulation videos and knowledge graphs of daily activities in a household" which is licensed under a Creative Commons Attribution 4.0 International License.
Based on the work available at https://github.com/aistairc/VirtualHome2KG.
This product is based on results obtained from a project, JPNP20006 and JPNP180013, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).