A project that allows to create and query a knowledge graph of videos and their DBpedia and Wikidata entities.
Entity linking allows leveraging information about named entities from a knowledge base.
This work shows that face recognition methods can be used to identify entities in videos such that users can formulate detailed queries with information from DBpedia and Wikidata. The focus of this project lies on videos from YouTube.
To address this issue we use publicity available thumbnails from DBpedia and Wikidata to create face-embeddings for a large number of celebrities using the Deepface library. Additionally, our project allows to scrape material from Google Images to represent entities in different scenes. Experiments have shown that recognition based on approximate k-nearest neighbors leads to the best balance between accuracy and runtime. Finally, the detected entities are used to build a knowledge graph based on the paper VidOnt: a core reference ontology for reasoning over video scenes which represents the links between videos, entities and timestamps. This gives the advantage of queries which include co-occurrences of actors or other video-related information. The figure below illustrates the structure of the generated graphs.
Using the YouTube Faces Database and YouTube Celebrities Face Tracking and Recognition Dataset we achieved an accuracy of 0.603 and 0.64 respectively. Scraping additional images improved the results to 0.85 and 0.826. Evaluation steps for the IMDb-Face and IMDb-Wiki are implemented and allow future performance tests.
Follow these steps:
- In the root folder create a ssh-key with an empty passphrase:
$ ssh-keygen -t rsa- Setup the cloud environment:
$ terraform init
$ terraform plan
$ terraform apply --auto-approve- Use the output IP-address to connect to the instance:
$ ssh-keygen -R IP_instance
$ ssh -i pub_key root@IP_instance- Run the app:
$ sudo docker run -v /mnt/face-hunter-bucket:/root/FACE-HUNTER/face-hunter-bucket --privileged -it facehunters/face-hunter /bin/bash
$ cd root/FACE-HUNTER
$ python cli.py ARGUMENTS- Destroy the instance when finished:
$ terraform destroy --auto-approveThe module can be imported and used directly in Python or by accessing the command line interface.
Example configurations are available under utils.
A detailed documentation can be found here.
This project started at the end of March 2021 as a Team Project at the University of Mannheim. The team consists of:
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.