add loss_mean_squared_per_channel

[arrufat]

This pull request adds support for a loss that computes independently on each plan of the output tensor.
This loss is useful to estimate keypoints (for pose estimation, etc).
The article that motivated me to add this kind of loss was Simple Baselines for Human Pose Estimation and Tracking.
I have successfully trained a network that replicates that article with this loss.

As a unit test, I added a small example that takes a matrix with 9 white dots as an input and outputs 9 channels, each one with a white dot if there was a dot at the associated position.

Sorry for the delay in making this PR... I couldn't think of a simple example (that can actually be used to learn how to use the loss)

[davisking]

Thanks for the PR :)

Just a few minor things and then it's ready to merge.

[arrufat]

I've updated with the requested changes. Your suggested name is more self-explanatory.

[arrufat]

My last commit fixed a build error on the windows checks (it built fine on my machine with GCC-9.1).
Is there anything else I need to do?

[davisking]

Thanks, the code looks good. I tried running the test and it passes, but I notice the loss only drops a little. Is it possible to make the test so that it runs to a low loss? Otherwise it's not obvious that it's working. For instance, you could ask it to do something simple like train 10 linear functions rather than use this more complex network in the test now.

[davisking]

For instance, make up 10 random linear functions. Then use them to make training data. E.g. make a 10x5 random matrix w, a random 5x1000 random matrix x and set y to w*x. Train a network with just one linear layer to map x to y. It should be able to learn it to basically 0 error and do so very quickly.

[arrufat]

Ok, I'll try that. I also noticed the loss drops only a little, and I think it's because predicting everything as 0 already gives a pretty low error (since there's only one pixel that would differ). I'll think of a more obvious test.

[davisking]

That’s better. But a little linear thing that dropped the loss really low would be nice.

The unit tests also aren’t a good way to do documentation. That kind of thing should go in a fully documented example program (you don’t have to make one for this though).

[arrufat]

Ok, I understand. I'll redo the tests then, probably during this weekend :)

Edit: I did this, because sometimes, I find myself reading the unit tests to better understand how to use some parts of the library.

[arrufat]

I've implemented the suggested linear test. I did not fiddle much with the parameters, but I'm not very happy with the results. Maybe learning that many random functions is not easy...

• error_before = 2.03315
• error_after = 0.216274

For the moment, I haven't removed the previous test. Please let me know what you think, or if I misunderstood your test.

[davisking]

The new tests look good. With some minor tweaks they will probably give really low error.

[arrufat]

Thank you for all the suggestions, I've simplified the MSE computation and reduced the dimensions of the data, it still does not converge to 0 as fast as I'd like:

• error_before: 2.95371
• error_after: 0.137677

[arrufat]

I'm suspecting maybe the tests are too random? Maybe I need to generate a more structured weight matrix w?

[davisking]

Don’t call set_max_num_epochs. That’s probably telling it to stop early.

[arrufat]

The only reason why I put the value of 160 is because it stops at epoch 157 (where it reaches a learning rate of 1e-7). I've also tried with smaller learning rate, but I get the same results.

[arrufat]

I've modified the network a little bit:

using net_type = loss_mean_squared_per_channel_and_pixel<num_channels,                                                                                                                                                               
                    extract<0, num_channels, 1, dimension,                                                                                                                                                                           
                    fc<num_outputs,                                                                                                                                                                                                  
                    relu<bn_fc<fc<500,                                                                                                                                                                                               
                    input<matrix<float>>>>>>>>;

Now, the errors look like this:

• error_before: 2.25673
• error_after: 0.0214044

[davisking]

Cool. That's good enough for me :)

Remove the non-simple version of the test and the PR is good to go.

[arrufat]

Great! Thank you! I'm probably more happy than you about this PR (my first serious one).
I wanted to contribute to dlib so badly!!

[davisking]

No problem, thanks for making the PR :)