Making features and neural networks jitable

[mravanelli]

1- Check if all the project is now jitable
2- Do the little changes to features and archictecture.py to make everything jitable
I will focus on that once the performance issue is fixed.

[pplantinga]

From Mirco:
For jitability I have the following proposal
1- we can jit only the subset of functions that give a major speed-up. For now, I saw a major speed up only for customized rnns (and I think there could be a speedup for all the functions containing long for loops). As you can see from the ligru in architecture.py this can be done with torch.jit.ScriptModule and the decoration flag
2- We can expect that only a subset of users might want to export the model in c++. This operation typically happens after training. In practice, the user can call sb.recover_if_possible(min_key="wer") and only after that users can make the model jitable. Let me check if this might work. If so, we can provide some examples to the users about "how to make your model jitable"

[TParcollet]

Can we ask the pytorch team for some insights on this part? It could be great to give a general expected performance improvement scale to the user w.r.t to specific operations. @pplantinga In fact, if we also are interested in seducing companies, the possibility to export c++ is quite important ... So it could be great to have an easy interface for that.

[pplantinga]

We discussed this at the Architecture meeting today. Some notes from the meeting:

• It is not currently possible to support both shape inference and jitability.
• The devs at PyTorch consider shape inference to be harmful, so they aren't planning to make changes so that both are possible.
• For the majority of users, shape inference will be a large benefit so we plan to support that by default rather than jitability.
• Mirco will write a document explaining how to export c++ (basically, train the model as usual, comment out about 2 lines per forward method used in the model, then export).

[TParcollet]

"The devs at PyTorch consider shape inference to be harmful" -> But this is what is so powerful about PyTorch ... But I see.

[mravanelli]

Tomorrow, I have my monthly meeting with the pytorch guys and I will update them of that. I agree with Titouan that jitability is important to attract companies and hopefully have more funds to continue deploying the toolkit. The point is that the current jit is very limited and we cannot support automatic shape inference and jit at the same time. The trade-off, would be to allow users to make the code jitable with few ad-hoc modifications that we can describe in the documentation.

…

On Mon, 4 May 2020 at 15:44, Parcollet Titouan ***@***.***> wrote: "The devs at PyTorch consider shape inference to be harmful" -> But this is what is so powerful about PyTorch ... But I see. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEA2ZVRD7GDHQCVZ3O2YAADRP4LKJANCNFSM4MTC4MUQ> .