We use the Emmental framework. Emmental is a framework for building multimodal multi-task learning systems. A key feature of Emmental is its task flow design where models are defined by the data flow through modules. By reusing modules in different tasks, you easy can extend your model to a multi-task setting.
We high encourage you to check out the Emmental docs and Emmental tutorials to understand the framework.