Video Coding for Machine

A curated list of VCM resources and approaches.

This list is maintained by: [IMRE] PKU (PI: Prof. Ling-Yu Duan) and [STRUCT] PKU (PI: Prof. Jiaying Liu)

Join VCM

• MPEG VCM info page and email list [Link]
• IEEE ICIP 2020 VCM Special Session [Call For Paper] [Submission Site]
  • Advanced Image/Video Coding for Machine and Human Vision (IEEE ICIP 2020 Special Session), Organizers: Jiaying Liu, Anthony Vetro, Ling-Yu Duan, Dong Liu.

Works Done by IMRE and STRUCT

Human-Machine Colloborative Coding

• Scalable Image Coding [Web] [PDF]
  • Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach (arXiv 2020), Yueyu Hu, Shuai Yang, Wenhan Yang, Ling-Yu Duan, Jiaying Liu.
• Emerging Coding Paradigm VCM [Web] [PDF]
  • An Emerging Coding Paradigm VCM: A Scalable Coding Approach Beyond Feature and Signal (arXiv 2020), Sifeng Xia, Kunchangtai Liang, Wenhan Yang, Ling-Yu Duan and Jiaying Liu.

Digital Retina

• Three FLow Model [Web] [PDF]
  • Towards digital retina in smart cities: A model generation, utilization and communication paradigm (IEEE ICME 2019), Y. Lou, L. Duan, Y. Luo, Z. Chen, T. Liu, S. Wang, and W. Gao.
• Collaborative Computing [Web] [PDF]
  • Toward intelligent visual sensing and low-cost analysis: A collaborative computing approach (IEEE VCIP 2019), Y. Bai, L. Duan, Y. Luo, S. Wang, Y. Wen, and W. Gao.
• Unified Infrastructure [Web] [PDF]
  • Front-end smart visual sensing and back-end intelligent analysis: A unified infrastructure for economizing the visual system of city brain (IEEE JCAS 2019), Y. Lou, L. Duan, S. Wang, Z. Chen, Y. Bai, C. Chen, and W. Gao.
• Knowledge Service [Web] [PDF]
  • Toward knowledge as a service over networks: A deep learning model communication paradigm (IEEE JSAC 2019), Z. Chen, L. Duan, S. Wang, Y. Lou, T. Huang, D. O. Wu, and W. Gao.

Deep Learning Based Video Coding

• Coarse-to-Fine Hyper-Prior [Web] [PDF]
  • Coarse-to-Fine Hyper-Prior Modeling for Learned Image Compression (AAAI 2020), Yueyu Hu, Wenhan Yang, Jiaying Liu.
• Progressive Rethinking Network for in-Loop Filtering [Web] [PDF]
  • Partition Tree Guided Progressive Rethinking Network for in-Loop Filtering of HEVC (IEEE ICIP 2019), Dezhao Wang, Sifeng Xia, Wenhan Yang, Yueyu Hu, Jiaying Liu.
• Deep Inter Prediction [Web] [Code] [PDF]
  • Deep Inter Prediction via Pixel-Wise Motion Oriented Reference Generation (IEEE ICIP 2019), Sifeng Xia, Wenhan Yang, Yueyu Hu, Jiaying Liu.
• Deep Intra Prediction [Web] [Code] [PDF]
  • Progressive Spatial Recurrent Neural Network for Intra Prediction (IEEE TMM 2019), Yueyu Hu, Wenhan Yang, Mading Li, and Jiaying Liu.

Feature Compression

• CDVA [Web] [PDF]
  • Compact descriptors for video analysis: The emerging MPEG standard (IEEE TMM 2019), L.-Y. Duan, Y. Lou, Y. Bai, T. Huang, W. Gao, V. Chandrasekhar, J. Lin, S. Wang, and A. C. Kot.
• CDVS [Web] [PDF]
  • Overview of the MPEG-CDVS standard (IEEE TIP 2016), L.-Y. Duan, V. Chandrasekhar, J. Chen, J. Lin, Z. Wang, T. Huang, B. Girod, and W. Gao.

A Comprehensive List of Related Work

Human-Machine Colloborative Coding

• Scalable Facial Image Compression [Web] [Code] [PDF]
  • Scalable Facial Image Compression with Deep Feature Reconstruction (IEEE ICIP 2019), Shurun Wang, Shiqi Wang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Wen Gao.
• Detection-driven Image Compression [Web] [Code] [PDF]
  • Beyond Coding: Detection-driven Image Compression with Semantically Structured Bit-stream (PCS 2019), Tianyu He, Simeng Sun, Zongyu Guo, Zhibo Chen.
• High efficiency compression for object detection [Web] [Code] [PDF]
  • High efficiency compression for object detection (arXiv 2019), Hyomin Choi and Ivan V. Bajic.

Scalable Coding

• Hierarchical Feature Decorrelation [Web] [Code] [PDF]
  • Deep Scalable Image Compression via Hierarchical Feature Decorrelation (PCS 2019), Zongyu Guo, Zhizheng Zhang, Zhibo Chen.
• Joint Feature and Texture Coding [Web] [Code] [PDF]
  • Joint Feature and Texture Coding: Towards Smart Video Representation via Front-end Intelligence (IEEE TCSVT 2018), Siwei Ma, Xiang Zhang, Shiqi Wang, Xinfeng Zhang, Chuanmin Jia, Shanshe Wang.

Feature Compression

• Data-driven interest point selection [Web] [Code] [PDF]
  • Data-driven lightweight interest point selection for large-scale visual search (IEEE TMM 2018), F. Gao, X. Zhang, Y. Huang, Y. Luo, X. Li, and L.-Y. Duan.

Deep Learning Based Video Coding

• Survey and Case Study [Web] [PDF]
  • Deep learning-based video coding: A review and a case study (ACM Computing Surveys 2020), Dong Liu, Yue Li, Jianping Lin, Houqiang Li, Feng Wu.
• Review [Web] [PDF]
  • Image and Video Compression with Neural Networks: A Review (IEEE TCSVT 2019), Siwei Ma, Xinfeng Zhang, Chuanmin Jia, Zhenghui Zhao, Shiqi Wang, Shanshe Wang.
• Content-aware In-Loop Filter [Web] [PDF]
  • Content-aware convolutional neural network for in-loop filtering in high efficiency video coding (IEEE TIP 2019), Chuanmin Jia, Shiqi Wang, Xinfeng Zhang, Shanshe Wang, Jiaying Liu, Shiliang Pu, Siwei Ma.
• Joint Spatial-Temporal Correlation Exploration [Web] [PDF]
  • Learned Video Compression via Joint Spatial-Temporal Correlation Exploration (AAAI 2020), Haojie Liu, Han Shen, Lichao Huang, Ming Lu, Tong Chen, and Zhan Ma.
• Neural Compression + Non-Local Attention [Web] [PDF]
  • Neural Image Compression via Non-Local Attention Optimization and Improved Context Modeling (arXiv 2019), T Chen, H Liu, Z Ma, Q Shen, X Cao, Y Wang.
• Multi-Frame Priors [Web] [PDF] [Web] [PDF]
  • Learned Quality Enhancement via Multi-Frame Priors for HEVC Compliant Low-Delay Applications (arXiv 2019), M Lu, M Cheng, Y Xu, S Pu, Q Shen, Z Ma.
• Extreme Image Compression [Web] [PDF] [Web] [PDF]
  • Extreme Image Compression via Multiscale Autoencoders With Generative Adversarial Optimization (arXiv 2019), C Huang, H Liu, T Chen, S Pu, Q Shen, Z Ma.
• Gated Context Model [Web] [PDF] [Web] [PDF]
  • Gated Context Model with Embedded Priors for Deep Image Compression (arXiv 2019), H Liu, T Chen, P Guo, Q Shen, Z Ma.
• iWave [Web] [PDF]
  • iWave: CNN-based wavelet-like transform for image compression (IEEE TMM 2020), Haichuan Ma, Dong Liu*, Ruiqin Xiong, Feng Wu.
• Responses to Joint Call for Proposals [Web] [PDF]
  • Deep learning-based technology in responses to the joint call for proposals on video compression with capability beyond HEVC (IEEE TCSVT 2020), Dong Liu, Zhenzhong Chen, Shan Liu, Feng Wu.
• Frank-Wolfe Network [Web] [PDF]
  • Frank-Wolfe network: An interpretable deep structure for non-sparse coding (IEEE TCSVT 2020), Dong Liu, Ke Sun, Zhangyang Wang, Runsheng Liu, Zheng-Jun Zha.
• Arithmetic Coding [Web] [PDF]
  • Convolutional neural network-based arithmetic coding for HEVC intra-predicted residues (IEEE TCSVT 2020), Changyue Ma, Dong Liu, Xiulian Peng, Li Li, Feng Wu.
• Quadtree-Based Coding Framework [Web] [PDF]
  • Quadtree-based coding framework for high density camera array based light field image (IEEE TCSVT 2020), Li Li, Zhu Li, Bin Li, Dong Liu, Houqiang Li.
• GAN Intra Prediction [Web] [PDF]
  • Generative Adversarial Network Based Intra Prediction for Video Coding (IEEE TMM 2020), Linwei Zhu, Sam Kwong, Yun Zhang, Shiqi Wang, Xu Wang.

Spike Coding

• Intelligent Driving [Web] [Code] [PDF]
  • Spike coding for dynamic vision sensor in intelligent driving (IEEE Internet of Things Journal 2019), S. Dong, Z. Bi, Y. Tian, and T. Huang.
• Inter-Spike Intervals [Web] [Code] [PDF]
  • An efficient coding method for spike camera using inter-spike intervals (DCC 2019), S. Dong, L. Zhu, D. Xu, Y. Tian, and T. Huang.
• Lossy Compression [Web] [Code] [PDF]
  • Spike coding: Towards lossy compression for dynamic vision sensor (DCC 2019), Y. Fu, J. Li, S. Dong, Y. Tian, and T. Huang.