[Answered] Does Theseus support pruning width of network?

[jiezhangGt]

Hi, thanks for your share of Theseus. To my understanding, Theseus can support prune depth of the network， I wonder if it can also support prune width of the network?

[JetRunner]

Thanks for your interest in our work.

Well, we've discussed this in the last section of our paper.

By now, we can only reduce the depth of a model but we can also use a totally different network as a substitution (e.g., Reformer).

We also proposed to use an MLP to map between different hidden sizes, like the one in TinyBERT. We haven't tried that so far but we encourage the community for further improvement.

[JetRunner]

Closing it for being inactive. Pls. reopen it if you have further questions.