You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, our model implements an inform_model function to get task_id from the batch in this part of our code.
The current task_id is included in the batch here.
In the inference stage, or we say downstream adapting stage, the model re-initialize an only router here(i.e. with only 1 task) and always set the task_id to 0 (and that is why in modeling_bert the task_id is by default 0 so that we do not need to explicitly operate the task_id). Optimizing this router (or along with the prompts) yields a downstream composition of the pretrained prompts.
btw, if we have some correlation prior in between the downstream task and the pretraining tasks, we can also initialize the router with a pretrained router. In this case, the task_id is also 0.
In the pre-train stage, the sample should use corresponding task router (using task_id param) to get fused prompt embedding, but in the code, I found task_id set to 0 always in https://github.com/Hzfinfdu/MPMP/blob/master/Deep/modeling_bert.py#L548, is that any thing wrong with it ?
by the way, in the inference stage, what task_id number should the new tasks (e.g. unseen task category) use ?
The text was updated successfully, but these errors were encountered: