Please consider adding MIG (MI-rror with G-radient modification) to torch.nn

[YagaoDirac]

🚀 The feature, motivation and pitch

https://github.com/YagaoDirac/Pytorch-extension-from-YagaoDirac-v2/blob/main/v2%20with%20basic%20test.py
I implemented this 2 weeks ago. It's probably a better implementation of the Linear layer. It speeds up the training while let people stack much more such layers directly without any trick.

Alternatives

In the code I implemented 3 different types of similar purpose. Each of them are tested and can work individually( if my tests are not too wrong).

Additional context

Also, if you decide to add this to pytorch, remember to rename it.
Notice the "untested workflow" in the file. It's probably a better way, and can be integreted into the layer itself.
More info in the file.

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki

[YagaoDirac]

@albanD Hi, thank you for tagging this issue with "needs research". Finally someone trusts me at least a bit. If you need more info about the code, my twitter, github, gmail and discord, are all the same name. Simply dm me somewhere.

[mikaylagawarecki]

Hi @YagaoDirac , could you provide more detail on what MIG is (e.g. research papers where it is used or proposed) and further elaboration on what problem it is meant to solve?

[YagaoDirac]

Hi @YagaoDirac , could you provide more detail on what MIG is (e.g. research papers where it is used or proposed) and further elaboration on what problem it is meant to solve?

Hi @mikaylagawarecki . I'm glad people don't ignore my work.
Short answer: I explained it in the python file.

I'm a hobbist. I didn't write a paper for it since I'm recently too busy on some other tasks. MIG and 2 other tools in the code is designed as a better implementation of the FCNN(torch.nn.Linear). It's trainable no matter you stack however many of it in a row. (If you stack 5 FCNN in a row, the training is too slow. Basically I don't do it. But mig and 2 other tools can do the same thing with at least 10 stacked in a row.) According to my tests(if they are not too wrong), it's 1000x to 100_000 times faster in some cases. The test code is also in the code in the link.
Now I'm asking 2 of my friends to coop on a paper for this but they can not begin at the moment. One of them has to wait until sep this year, the other one is bit slower. So if Pytorch team is interested in this, you guys are absolutely much faster, so people will begin to know this tool earlier, which is great.

[YagaoDirac]

Update. I almost forgot this thread.
If you can wait for my paper, you don't need to read this post.
After I recheck(probably in the late 2024), I found the "mirror" doesn't work. The "gramo" works. If you are using this, check out my latest code. It's already corrected there.
I plan to write the paper in late this year(2025). I'm on some other jobs recently.
Have a nice day.