Skip to content

An intelligent multimodal-learning based system for video, product and ads analysis. Based on the system, people can build a lot of downstream applications such as product recommendation, video retrieval, etc.

hysia_v2
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Hysia Video to Retail Platform

* This project is supported by Cloud Application and Platform Lab led by Prof. Yonggang Wen

codebeat badge

An intelligent multimodal-learning based system for video, product and ads analysis. You can build various downstream applications with the system, such as product recommendation, video retrieval. Several examples are provided.

The system is under active development currently. You are welcome to create a issue, pull request here. We will credit them into our next version.

hysia-block-diagram

ShowcaseFeaturesSetup EnvironmentConfigurationDemoCitation

News

  • (2020-08) The work has been accepted as an open-source competation paper at ACMMM2020!
  • (2020-05) The docker image has been updated
  • (2020-05) You can easily bind your model to our system

Showcase

👉 Full list of showcase.

  1. Upload video and process it by selecting different models

    select-models

  2. Display video processing result

    display-analytic-result

  3. Search scene by image and text

    search-result

  4. Insert product advertisement and display insertion result

    view-ads

Features

  • Multimodal learning-based video analysis:
    • Scene / Object / Face detection and recognition
    • Multimodality data pre-processing
    • Results align and store
  • Downstream applications:
    • Intelligent ads insertion
    • Content-product match
  • Visualized testbed
    • Visualize multimodality results
    • Can be installed separately

Setup Environment

1. Download Data

👉 For no Google Drive access.

# Make sure this script is run from project root
bash scripts/download-data.sh

2. Installation

👉 Install with Docker 🐳

docker pull hysia/hysia:v2o

Configuration

Change decoder and model server running devices at device_placement.yml:

decoder: CPU
visual_model_server: CUDA:1
audio_model_server: CUDA:2
feature_model_server: CUDA:3
product_search_server: CUDA:2
scene_search_server: CUDA:3

Device value format: cpu, cuda or cuda:<int>.

Demo

Run with docker 🐳

docker run --rm \
  --gpus all -d -p 8000:8000 \
  -v ${PWD}/server/config/device_placement.yml:/content/server/config/device_placement.yml \
  hysia/hysia:v2o

Then you can go to http://localhost:8000. Use username: admin and password: admin to login.

Some Useful Tools

  • Large dataset preprocessing
  • Video/audio decoding
  • Model profiling
  • Multimodality data testbed

Contributing

You are welcome to contribute to Hysia! Please refer to here to get start.

Paper Citation

If you use Hysia in your work, we would be very grateful if you cite

@inproceedings{10.1145/3394171.3414536,
    author = {Zhang, Huaizheng and Li, Yuanming and Ai, Qiming and Luo, Yong and Wen, Yonggang and Jin, Yichao and Ta, Nguyen Binh Duong},
    title = {Hysia: Serving DNN-Based Video-to-Retail Applications in Cloud},
    year = {2020},
    booktitle = {Proceedings of the 28th ACM International Conference on Multimedia},
    pages = {4457–4460},
}

About Us

Maintainers

  • Huaizheng Zhang :octocat:
  • Yuanming Li :octocat:
  • Qiming Ai :octocat:

Previous Contributors

  • Shengsheng Zhou :octocat:
  • Wenbo Jiang (Now, Shopee) :octocat:
  • Ziyuan Liu (Now, Tencent) :octocat:
  • Yongjie Wang (Now, NTU PhD) :octocat:

About

An intelligent multimodal-learning based system for video, product and ads analysis. Based on the system, people can build a lot of downstream applications such as product recommendation, video retrieval, etc.

Topics

Resources

License

Releases

No releases published

Packages

No packages published