implement TripletMarginLoss as a native function #5680

li-roy · 2018-03-09T22:33:16Z

Implement TripletMarginLoss as a native function
add size_average arg
add reduce arg as per losses per-batch-element #264

Benchmarks run on inputs of size (5,500) x 5000 iterations.

old:
forward [1.0334448860958219, 1.0698160571046174, 1.054812144022435]
backward [1.6976923400070518, 1.6186234278138727, 1.617640192154795]
double backward [3.2021269120741636, 3.165953448973596, 3.24088541790843]

new:
forward [0.8313469190616161, 0.8255189431365579, 0.8157028129789978]
backward [1.5887232730165124, 1.6056769208516926, 1.5178304109722376]
double backward [3.154403690015897, 3.1905114939436316, 3.2450901730917394]

torch/nn/modules/loss.py

@@ -959,16 +967,17 @@ class TripletMarginLoss(Module):
        http://www.iis.ee.ic.ac.uk/%7Evbalnt/shallow_descr/TFeat_paper.pdf
    """

-    def __init__(self, margin=1.0, p=2, eps=1e-6, swap=False):
-        super(TripletMarginLoss, self).__init__()
+    def __init__(self, margin=1.0, p=2, eps=1e-6, swap=False, size_average=True, reduce=True):


test/test_nn.py

+        constructor_args=(torch.rand(10),),
+        input_fn=lambda: torch.randn(5, 10),
+        target_fn=lambda: torch.rand(5, 10).mul(2).floor(),
+        reference_fn=lambda i, t, m: -((t * i.sigmoid().log() + (1 - t) * (-i).sigmoid().log()) * get_weight(m)).sum() /


aten/src/ATen/native/Distance.cpp

+Tensor pairwise_distance(const Tensor& x1, const Tensor& x2, double p, double eps) {
+  auto diff = abs(x1 - x2);
+  auto out = pow(diff + eps, p).sum(1);
+  return pow(out, 1 / p);


goldsborough · 2018-03-14T04:17:53Z

@pytorchbot retest this please

zou3519

Code looks good to me. I had a few minor comments and a concern that we're subtly changing the behavior of pairwise_distance

torch/nn/modules/loss.py

+        margin (float, optional): Default: `1`.
+        p (int, optional): The norm degree for pairwise distance. Default: `2`.
+        swap (float, optional): The distance swap is described in detail in the paper
+            `Learning shallow convolutional feature descriptors with triplet losses` by


torch/nn/modules/loss.py

@@ -959,16 +967,17 @@ class TripletMarginLoss(Module):
        http://www.iis.ee.ic.ac.uk/%7Evbalnt/shallow_descr/TFeat_paper.pdf
    """

-    def __init__(self, margin=1.0, p=2, eps=1e-6, swap=False):
-        super(TripletMarginLoss, self).__init__()
+    def __init__(self, margin=1.0, p=2, eps=1e-6, swap=False, size_average=True, reduce=True):


aten/src/ATen/native/native_functions.yaml

@@ -271,6 +271,9 @@
 - func: ones_like(Tensor self, *, Type dtype) -> Tensor
  variants: function

+- func: pairwise_distance(Tensor x1, Tensor x2, double p, double eps) -> Tensor


aten/src/ATen/native/native_functions.yaml

@@ -360,6 +363,9 @@
 - func: transpose_(Tensor self, int64_t dim0, int64_t dim1) -> Tensor
  variants: method

+- func: triplet_margin_loss(Tensor anchor, Tensor positive, Tensor negative, double margin, double p, double eps, bool swap, bool size_average, bool reduce) -> Tensor


aten/src/ATen/native/Distance.cpp

+
+Tensor pairwise_distance(const Tensor& x1, const Tensor& x2, double p, double eps) {
+  auto diff = abs(x1 - x2 + eps);
+  return norm(diff, p, 1);


aten/src/ATen/native/Distance.cpp

+namespace at { namespace native {
+
+Tensor pairwise_distance(const Tensor& x1, const Tensor& x2, double p, double eps) {
+  auto diff = abs(x1 - x2 + eps);


torch/nn/functional.py

-    assert x1.size() == x2.size(), "Input sizes must be equal."
-    assert x1.dim() == 2, "Input must be a 2D matrix."
-    diff = torch.abs(x1 - x2)
-    out = torch.pow(diff + eps, p).sum(dim=1, keepdim=True)


torch/nn/functional.py

@@ -1960,11 +1960,7 @@ def pairwise_distance(x1, x2, p=2, eps=1e-6):
        >>> output = F.pairwise_distance(input1, input2, p=2)


zou3519

LGTM. Minor nit below

aten/src/ATen/native/Distance.cpp

@@ -5,7 +5,6 @@
 namespace at { namespace native {

 Tensor pairwise_distance(const Tensor& x1, const Tensor& x2, double p, double eps) {


zou3519

Yay! Approve approve approve

Roy Li added 3 commits March 9, 2018 14:27

implement TripletMarginLoss as a native function

fc9a4f8

implement TripletMarginLoss as native function

2913cec

fix compile error

cb69c3a

zou3519 reviewed Mar 12, 2018

View reviewed changes

aten/src/ATen/native/Distance.cpp Outdated

Tensor pairwise_distance(const Tensor& x1, const Tensor& x2, double p, double eps) {

auto diff = abs(x1 - x2);

auto out = pow(diff + eps, p).sum(1);

return pow(out, 1 / p);

This comment was marked as off-topic.

Sign in to view

address comments

4d23df1

zou3519 reviewed Mar 14, 2018

View reviewed changes

address comments

7354bb7

zou3519 approved these changes Mar 15, 2018

View reviewed changes

Add keepdim arg to pairwise distance

1f0560d

zou3519 approved these changes Mar 16, 2018

View reviewed changes

soumith merged commit e876b5d into pytorch:master Mar 17, 2018

li-roy deleted the triplet branch March 20, 2018 14:13

		@@ -1960,11 +1960,7 @@ def pairwise_distance(x1, x2, p=2, eps=1e-6):
		>>> output = F.pairwise_distance(input1, input2, p=2)

		@@ -5,7 +5,6 @@
		namespace at { namespace native {

		Tensor pairwise_distance(const Tensor& x1, const Tensor& x2, double p, double eps) {

implement TripletMarginLoss as a native function #5680

implement TripletMarginLoss as a native function #5680

Uh oh!

Conversation

li-roy commented Mar 9, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

goldsborough commented Mar 14, 2018

Uh oh!

zou3519 left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

zou3519 left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

zou3519 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!