Implement MarginRankingLoss as native function and add reduce=True arg to it

[li-roy]

• Implemented MarginRankingLoss as native function.
• As per losses per-batch-element #264. When reduce is False, MarginRankingLoss outputs a loss per sample in minibatch. When reduce is True (default), the current behavior is kept.

Test Plan
test/run_test.sh
Added unit tests for MarginRankingLoss.

[ssnl]

why doesn't this also have a backward function?

[li-roy]

I used #5080 as a reference for this PR, and a backwards wasn't implemented there as well. Should I also add a backwards?

[ssnl]

The fact that it is moved from python to cpp might be enough to justify ignoring backward. You should either implement it or do a speed benchmark as #5080 .

[ezyang]

17:48:04 ======================================================================
17:48:04 ERROR: test_MarginRankingLoss_reduce (__main__.TestNN)
17:48:04 ----------------------------------------------------------------------
17:48:04 Traceback (most recent call last):
17:48:04   File "test_nn.py", line 4576, in <lambda>
17:48:04     setattr(TestNN, test_name, lambda self, test=test: test(self))
17:48:04   File "/var/lib/jenkins/workspace/test/common_nn.py", line 1041, in __call__
17:48:04     deepcopy(target), module)
17:48:04   File "test_nn.py", line 4735, in <lambda>
17:48:04     marginrankingloss_reference(i[0], i[1], t, size_average=get_size_average(m)),
17:48:04 NameError: global name 'marginrankingloss_reference' is not defined
17:48:04 
17:48:04 ======================================================================
17:48:04 ERROR: test_MarginRankingLoss_reduce_no_size_average (__main__.TestNN)
17:48:04 ----------------------------------------------------------------------
17:48:04 Traceback (most recent call last):
17:48:04   File "test_nn.py", line 4576, in <lambda>
17:48:04     setattr(TestNN, test_name, lambda self, test=test: test(self))
17:48:04   File "/var/lib/jenkins/workspace/test/common_nn.py", line 1041, in __call__
17:48:04     deepcopy(target), module)
17:48:04   File "test_nn.py", line 4735, in <lambda>
17:48:04     marginrankingloss_reference(i[0], i[1], t, size_average=get_size_average(m)),
17:48:04 NameError: global name 'marginrankingloss_reference' is not defined
17:48:04 

[li-roy]

@ezyang Yeah I have an update, was going to run benchmarks first.

[li-roy]

benchmarks run with inputs of size (1000), 5000 times:

forward (old) [0.8390149101614952, 0.8477208158001304, 0.859797858633101]
backward (old) [2.4765465622767806, 2.535214721225202, 2.4746940098702908]
double backward (old) [0.7999639995396137, 0.7724319389089942, 0.7383095575496554]

forward (new) [0.7037791842594743, 0.7032447448000312, 0.6909415749832988]
backward (new) [0.8209385611116886, 0.8369621662423015, 0.8271905779838562]
double backward (new) [0.8673017444089055, 0.8709612749516964, 0.8717606142163277]

[li-roy]

@pytorchbot retest this please

[ssnl]

Interesting that double backward slows down, but this is fine given the single backward improvements.

[li-roy]

@pytorchbot retest this please

[li-roy]

@pytorchbot retest this please

[li-roy]

@pytorchbot retest this please

[ssnl]

Looks good in general. But please fix the conflict :)

[ssnl]

LGTM!