Add mul gradients

[vnghia]

@saxenasaurabh

Part of #42668

I would like to ask something:

• Should we file an issue to track the progress of this project ?
• Is there any timeline for this project in general ?
• The test is quite large. Should we seperate it into multiple files ( something like math_grad_test.cc and unified_api_math_test.py )

[vnghia]

@saxenasaurabh
Could you review this PR please ?

[saxenasaurabh]

Sorry for the hold up, I will review this today.

[saxenasaurabh]

Should we file an issue to track the progress of this project ?

That's an interesting idea and something I would definitely like to try. For now I have been tracking issues and progress using Google internal tools. Let me know if you have any suggestions on how we can manage this on github. @dynamicwebpaige may have pointers on managing OSS projects.

Is there any timeline for this project in general ?

I think this project overall will span into Q1'21. In Q4, I want to focus on fleshing out the C++ infrastructure. E.g. Accidental memory leaks are a big problem right now. We also need to make sure that C++ gradient functions are as easy to write as python gradients.

Also getting ResNet training using C++ gradients would be a great milestone! I had done some benchmarking for simple MLPs on MNIST and those were 2x faster in eager mode. If we can get similar gains on ResNet, that would be amazing I think.

I also want to figure out how these gradient functions can be used from python alongside other python gradients. I have this and other things written down in a doc. I will try to publish it externally asap.

The test is quite large. Should we seperate it into multiple files ( something like math_grad_test.cc and unified_api_math_test.py )

That's a great idea. We should try to colocate gradients tests with the gradients code.

[vnghia]

Let me know if you have any suggestions on how we can manage this on github

I don't know any tool honestly. I am just thinking that we could use a list of checkboxes to maintain the list of gradients but I did not think any further.