Transferring knowledge to my own dataset

[SUYEgit]

Hi, I read your paper recently and it's really brilliant!

Now I have a dataset labelled with object classes and semantic segmentation mask, say this is the target task on my dataset. The classes will be different from your pre-trained models. I want to do transfer learning using your task-bank as source task(taking the encoder parts) and train my model. Could you please tell me how to do that?

[amir32002]

Please see the procedure described in section "3. Method" (look for definition of "target-only" tasks) and section "4.2. Generalization to Novel Tasks" of the paper. You can follow the same process to transfer to your own task/dataset. You can use the released task bank models as the source encoders, then you'll need to train the small transfer functions (see section 3.2) to quantify the transfer performance and pick the best source(s).

[emergencyd]

Hi! I'm really interested in your brilliant work!
I'm wondering, if I want to transfer your model to mine, how can I choose the appropriate source tasks. Also, I've noticed that, in your experiment on MIT Places & ImageNet, you choose data with similar classes as yours. But what if I want to do semantic segmentation that has totally different classes as yours.
Thanks!!

[amir32002]

Thanks for your interest. The reply above answers your question as well. The process of picking appropriate sources tasks for a novel target tasks is described there.

The experiments on MIT Places and ImageNet in the paper are about evaluating dataset universality, not novel tasks, so they're not related to novel tasks or semantic classes. In the formulation, there is no difference between a novel non-semantic task, a semantic task with novel classes, etc -- any novel task is treated the same way by the framework.

Please see the procedure described in section "3. Method" (look for definition of "target-only" tasks) and section "4.2. Generalization to Novel Tasks" of the paper. You can follow the same process to transfer to your own task/dataset. You can use the released task bank models as the source encoders, then you'll need to train the small transfer functions (see section 3.2) to quantify the transfer performance and pick the best source(s).

[emergencyd]

Thank you for your answer! Just want to make sure that I understand it clearly. Assume that my target task is task X with dataset D1, and your dataset is D0. Now I'm going to transfer to my novel task with novel dataset.

1. step 1: Design transfer function
2. step 2: Concatenate transfer function with your pretrained models (based on D0), to generate a bunch of transfers
3. step 3: Train the transfer function based on D1
4. step 4: Set my maximum order and supervision budget. Use AHP and BIP to form the subgraph, where taskonomy tasks are source-only tasks, and mine is target-only task.

After that, when I want to solve task X with any dataset D, I can just pick the desired pretrained models from your taskbank, then I only need to train the transfer function using dataset D.

[amir32002]

I describe a way for transferring to your own dataset for your novel task below, but note that Taskonomy is concerned with task space and studies how tasks transfer to each other. How different datasets are involved in this process is not a primary subject in this study (e.g. in your case where you not only have a novel task, but also a new dataset for that task) but I describe one practical approach below. Other ways are conceivable too.

If you just had a novel task (not a new dataset) and wanted to see what sources best transfer to it: the most concrete way would be to pick the small subset of our dataset on which we trained on transfer function and annotate them for your new task. Then you can just treat that as a target-only task and follow the rest of the standard process of taskonomy (train transfer function, do AHP, then do BIP). This will ensure that all images seen by all tasks and transfers are identical, hence the image distribution would not make an impact in the conclusion of what source tasks best transfer to your novel tasks. (see "Dataset" discussion in Sec. 3 of the paper).

However, you not only have a novel task, you also have a new dataset for it, and I assume you wish to avoid doing any annotation. Then you can pick the encoder neural networks we released in the Task Bank repository and train transfer functions from their internal representation to your novel task using the training split of your own dataset. Then you should quantify their transfer performance using the validation split of your dataset. That gives you something akin to one row of the affinity matrix we reported in the paper. If you have only one novel task, you dont need AHP or BIP. You can simply pick the best performing transfer function (and its corresponding source tasks) as the best transfer policy. AHP and BIP are not needed since you have only one novel task you're interested in, hence there is no need to normalize the performance using AHP wrt to any other task, and also no need to solve a subgraph selection problem and constraint satisfaction, since the solution would be trivially the same as taking the argmax of transfer_performances. However, if you have more than one novel task and wish Taskonomy method to optimize what collection of source tasks best support your entire novel task set while minimizing the required source tasks and their order, then you need to run AHP followed by BIP to have the best transfer policy.