Awesome Multi-Task Learning

This page contains a list of papers on multi-task learning for computer vision. Please create a pull request if you wish to add anything. If you are interested, consider reading our recent survey paper.

Multi-Task Learning for Dense Prediction Tasks: A Survey

Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai and Luc Van Gool.

Workshop

📢 📢 📢 We are organizing a workshop on multi-task learning at ICCV 2021. More information can be found on our website.

• April 10: We have confirmed eight excellent speakers, including Rich Caruana (Microsoft), Chelsea Finn (Stanford), Judy Hoffman (Georgia Tech), Iasonas Kokkinos (University College London), Andrew Rabinovich (Headroom inc.), Raquel Urtasun (University of Toronto), Luc Van Gool (Ku Leuven & ETH Zurich) and Amir Zamir (EPFL).
• June 2: Submission website is now live.

Table of Contents:

• Survey papers
• Datasets
• Architectures
  • Encoder-based
  • Decoder-based
  • Other
• Neural Architecture Search
• Optimization strategies
• Transfer learning

Survey papers

• Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., & Van Gool, L. Multi-Task Learning for Dense Prediction Tasks: A Survey, T-PAMI, 2020. [PyTorch]

• Ruder, S. An overview of multi-task learning in deep neural networks, ArXiv, 2017.

• Zhang, Y. A survey on multi-task learning, ArXiv, 2017.

• Gong, T., Lee, T., Stephenson, C., Renduchintala, V., Padhy, S., Ndirango, A., ... & Elibol, O. H. A comparison of loss weighting strategies for multi task learning in deep neural networks, IEEE Access, 2019.

Datasets

The following datasets have been regularly used in the context of multi-task learning:

• NYUDv2
• Cityscapes
• PASCAL
• Taskonomy
• KITTI
• SUN RGB-D
• BDD100K

Architectures

Encoder-based architectures

• Misra, I., Shrivastava, A., Gupta, A., & Hebert, M. Cross-stitch networks for multi-task learning, CVPR, 2016. [PyTorch]

• Gao, Y., Ma, J., Zhao, M., Liu, W., & Yuille, A. L. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction, CVPR, 2019. [Tensorflow] [PyTorch]

• Liu, S., Johns, E., & Davison, A. J. End-to-end multi-task learning with attention, CVPR, 2019. [PyTorch]

Decoder-based architectures

• Xu, D., Ouyang, W., Wang, X., & Sebe, N. Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing, CVPR, 2018.

• Zhang, Z., Cui, Z., Xu, C., Jie, Z., Li, X., & Yang, J. Joint task-recursive learning for semantic segmentation and depth estimation, ECCV, 2018.

• Ruder, S., Bingel, J., Augenstein, I., & Søgaard, A. Latent multi-task architecture learning, AAAI, 2019.

• Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., & Yang, J. Pattern-affinitive propagation across depth, surface normal and semantic segmentation, CVPR, 2019.

• Zhou, L., Cui, Z., Xu, C., Zhang, Z., Wang, C., Zhang, T., & Yang, J. Pattern-Structure Diffusion for Multi-Task Learning, CVPR, 2020.

• Vandenhende, S., Georgoulis, S., & Van Gool, L. MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning, ECCV, 2020. [PyTorch]

Other

• Yang, Y., & Hospedales, T. Deep multi-task representation learning: A tensor factorisation approach, ICLR, 2017.

• Kokkinos, Iasonas. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory, CVPR, 2017.

• Rebuffi, S. A., Bilen, H., & Vedaldi, A. Learning multiple visual domains with residual adapters, NIPS, 2017.

• Long, M., Cao, Z., Wang, J., & Philip, S. Y. Learning multiple tasks with multilinear relationship networks, NIPS, 2017.

• Meyerson, E., & Miikkulainen, R. Beyond shared hierarchies: Deep multitask learning through soft layer ordering, ICLR, 2018.

• Rosenbaum, C., Klinger, T., & Riemer, M. Routing networks: Adaptive selection of non-linear functions for multi-task learning, ICLR, 2018.

• Mallya, A., Davis, D., & Lazebnik, S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights, ECCV, 2018.

• Rebuffi, S. A., Bilen, H., & Vedaldi, A. Efficient parametrization of multi-domain deep neural networks, CVPR, 2018.

• Maninis, K. K., Radosavovic, I., & Kokkinos, I. Attentive single-tasking of multiple tasks, CVPR, 2019. [PyTorch]

• Kanakis, M., Bruggemann, D., Saha, S., Georgoulis, S., Obukhov, A., & Van Gool, L. Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference, ECCV, 2020.

• Wang, Q., Ke, J., Greaves, J., Chu, G., Bender, G., Sbaiz, L., Go, A., Howard, A., Yang, F., Yang, M.H. & Gilbert, J. Multi-path Neural Networks for On-device Multi-domain Visual Classification, WACV, 2021.

• Bruggemann, D., Kanakis, M., Obukhov, A., Georgoulis, S., & Van Gool, L. Exploring Relational Context for Multi-Task Dense Prediction, ArXiv, 2021.

Neural Architecture Search

• Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., & Feris, R. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification, CVPR, 2017.

• Bragman, F. J., Tanno, R., Ourselin, S., Alexander, D. C., & Cardoso, J. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels, ICCV, 2019.

• Newell, A., Jiang, L., Wang, C., Li, L. J., & Deng, J. Feature partitioning for efficient multi-task architectures, ArXiv, 2019.

• Guo, P., Lee, C. Y., & Ulbricht, D. Learning to Branch for Multi-Task Learning, ICML, 2020.

• Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., & Savarese, S. Which Tasks Should Be Learned Together in Multi-task Learning?, ICML, 2020.

• Vandenhende, S., Georgoulis, S., De Brabandere, B., & Van Gool, L. Branched multi-task networks: deciding what layers to share, BMVC, 2020.

• Bruggemann, D., Kanakis, M., Georgoulis, S., & Van Gool, L. Automated Search for Resource-Efficient Branched Multi-Task Networks, BMVC, 2020.

• Sun, X., Panda, R., & Feris, R. AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning, NIPS, 2020.

Optimization strategies

• Kendall, A., Gal, Y., & Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, CVPR, 2018.

• Zhao, X., Li, H., Shen, X., Liang, X., & Wu, Y. A modulation module for multi-task learning with applications in image retrieval, ECCV, 2018.

• Chen, Z., Badrinarayanan, V., Lee, C. Y., & Rabinovich, A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks, ICML, 2018.

• Sener, O., & Koltun, V. Multi-task learning as multi-objective optimization, NIPS, 2018. [PyTorch]

• Liu, P., Qiu, X., & Huang, X. Adversarial multi-task learning for text classification, ACL, 2018.

• Guo, M., Haque, A., Huang, D. A., Yeung, S., & Fei-Fei, L. Dynamic task prioritization for multitask learning, ECCV, 2018.

• Lin, X., Zhen, H. L., Li, Z., Zhang, Q. F., & Kwong, S. Pareto multi-task learning, NIPS, 2019.

• Suteu, M., & Guo, Y. Regularizing Deep Multi-Task Networks using Orthogonal Gradients, ArXiv, 2019.

• Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. Gradient surgery for multi-task learning, NIPS, 2020. [Tensorflow]

• Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., & Anguelov, D. Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout, NIPS, 2020.

• Li, W. H., & Bilen, H. Knowledge Distillation for Multi-task Learning, ECCV-Workshop, 2020. [PyTorch]

• Borse, S., Wang, Y., Zhang, Y., & Porikli, F. InverseForm: A Loss Function for Structured Boundary-Aware Segmentation, CVPR 2021.

• Vasu P., Saxena S., Tuzel O. Instance-Level Task Parameters: A Robust Multi-task Weighting Framework, ArXiv, 2021.

Transfer learning & Domain Adaptation

• Cui, Y., Song, Y., Sun, C., Howard, A., & Belongie, S. Large scale fine-grained categorization and domain-specific transfer learning, CVPR, 2018.

• Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., & Savarese, S. Taskonomy: Disentangling task transfer learning, CVPR, 2018. [PyTorch]

• Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., ... & Perona, P. Task2vec: Task embedding for meta-learning, ICCV, 2019. [PyTorch]

• Dwivedi, K., & Roig, G. Representation similarity analysis for efficient task taxonomy & transfer learning, CVPR, 2019. [PyTorch]

• Saha, S., Obukhov, A., Paudel, D. P., Kanakis, M., Chen, Y., Georgoulis, S., & Van Gool, L. Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation, CVPR, 2021.

Robustness

• Mao, C., Gupta, A., Nitin, V., Ray, B., Song, S., Yang, J., & Vondrick, C. Multitask Learning Strengthens Adversarial Robustness, ECCV, 2020. [PyTorch]

• Zamir, A. R., Sax, A., Cheerla, N., Suri, R., Cao, Z., Malik, J., & Guibas, L. J. Robust Learning Through Cross-Task Consistency, CVPR, 2020.

• Georgescu, M. I., Barbalau, A., Ionescu, R. T., Khan, F. S., Popescu, M., & Shah, M. Anomaly Detection in Video via Self-Supervised and Multi-Task Learning, CVPR, 2021.