Semi-hard negative bug?

[melgor]

@bamos
I recently read the semi-hard negative code. I am not sure, but I think it may not work as assumed.
I am talking about that fragment from traininig/train.lua

if posDist < negDist and negDist < selNegDist and negDist < alpha then
                selNegDist = negDist
                selNegIdx = k
 end

The main concert is connected with negDist < alpha
Alpha is set to 0.2 and negDist would be never lower than that (and in the same time bigger than posDist), I think that condition is wrong.
I make a quick test and never this condition was satisfied.

Suggestion 1

I think, that based on TripletLoss, the condition should looks like that:

if posDist < negDist and negDist < selNegDist and math.abs(posDist-negDist) < alpha then

It choose that negative example which distance is higher than from positive one, but in other hand, no further than alpha.

Suggestion 2

But, after some test, I think that this condition should be better. Sometimes for person no negative example satisfy the conditions, then it is chosen the random negative example. I am proposing sth like that:

if posDist < negDist and negDist < selNegDist then 
          selNegDist = negDist 
          selNegIdx = k 
          if math.abs(posDist-negDist) < alpha then 
            break 
          end 
end

It take negative example, which are better than previous one and additionally break when "alpha" condition is satisfied. I think that we do not need to have best negative example, which satisfy condition (this is similar way to Oxford-Face algorithm for Triplet selection). Additional, it will speed up all process.

But first of all, am I right claiming that current semi-hard negative condition does not work as should and now it choose only random negative example?

[bamos]

Hi @melgor - thanks for reporting this!
The negDist < alpha is indeed inconsistent with the FaceNet paper.

I've updated the code to your second suggestion and added some output info about how many random
negatives are selected in the offline processing. Also posted this to the mailing list at https://groups.google.com/forum/#!topic/cmu-openface/4yk9dO1Z0ww

Interestingly the conditions with negDist < alpha is satisfied about half the time in the early phases of training, which is what I looked at when I validated this portion of code. For the same random seed and images, Suggestion 1 gives 513 random triplets out of 1006 total triplets, Suggestion 2 gives 490, and the bug (negDist < alpha) gives 637.

I tested this for later iterations with the nn4.v1 model and the bug is much closer to random as you mention. Suggestion 1 gives 498 random triplets out of 996 total, Suggestion 2 also gives 498, and the bug gives 974!

I've been training a new model with suggestion 1 over the past few days, interested to see how much this improves the accuracy.

-Brandon.

[bamos]

Re-opening, I'm having some strange accuracy issues with this. @melgor - have you successfully trained a model with your Suggestion 2?

In the following, I switched from your Suggestion 1 to suggestion 2 at about iteration 45. I don't yet understand why suggestion 2 causes such bad performance. Any ideas?

Since suggestion 1's model was starting to perform better, I've made this the default version for now.

Iter 40

• 0.73 avg LFW accuracy

Latest Iter

• 0.51 avg LFW accuracy.

[melgor]

It it really strange. I have one idea why it may work so bad, but still it
will not explain the random guess. I must look it closer.

About my tests, I do not have 12 GB on my card, so I only test 2-step
training (like in Oxford Face paper). And here the result using all three
different sampling strategy are similar and does not produce better feature
representstion than original net. So sth is wrong here.

I think that you should stay at Suggestion 1, it may be better is some
cases.

9:58 PM pon., 09.11.2015 Brandon Amos użytkownik notifications@github.com
napisał:

Re-opening, I'm having some strange accuracy issues with this. @melgor
https://github.com/melgor - have you successfully trained a model with
your Suggestion 2?

In the following, I switched from your Suggestion 1 to suggestion 2 at
about iteration 45. I don't yet understand why suggestion 2 causes such bad
performance. Any ideas?

Since suggestion 1's model was starting to perform better, I've made this
the default version for now.

[image: loss]
https://cloud.githubusercontent.com/assets/707462/11046106/631a61e6-86f9-11e5-8735-ed6f87b00f15.png
Iter 40

• 0.73 avg LFW accuracy

[image: roc]
https://cloud.githubusercontent.com/assets/707462/11046189/ecfe92b0-86f9-11e5-8d99-ef577e36bb37.png
Latest Iter

• 0.51 avg LFW accuracy.

[image: roc]
https://cloud.githubusercontent.com/assets/707462/11046103/5f4e8286-86f9-11e5-83e2-9987d73330ff.png

—
Reply to this email directly or view it on GitHub
#48 (comment)
.

[bamos]

@melgor - suggestion 1 is working well so far. Averaging ~300 random triplets out of 1000 in the offline sampling and almost at the previous net's accuracy.

Epoch 55

• accuracy: 0.7773
  

Epoch 54

• accuracy: 0.7915
  

[bamos]

Training with suggestion 1 is clearly faster as the experiment from the past week shows, closing this issue. The accuracy hasn't surpassed the existing models, but I'm going to keep the training going.

LFW Experiment, Epoch 142

• Accuracy: 0.8390