Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TMVA] Add new Evaluation Metric ( meanAbsoluteError between two matrices ) #2376

Merged
merged 1 commit into from Aug 2, 2018

Conversation

4 participants
@ravikiran0606
Copy link
Contributor

commented Jul 27, 2018

Need:

The need for a new evaluation metric for testing the convergence of the optimizer is essential. The already existing metric was maximumRelativeError() between two matrices which takes the maximum of all the relative errors between its individual elements. But the relative error between these elements depends on the element values. i.e

Relative error between a and b = abs(a-b)/(abs(a)+abs(b)). Let use consider 2 cases,

case a) If two values are a = 0.0001 , b = 0.0002, relative error = 0.3333
case b) If two values are a = 10.0001 b = 10.0002 relative error = 4.99992e-6

Since the unit tests for optimizer is written in a way so that a sample 3 layer DNN will learn this function Y = K * X. So, If X = I ( Identity matrix ), then Y = K * I = K. This should be equivalent to the output of the trained DNN when I is feed as Input. Let Y' be the output of the trained DNN. So I need to compare the matrices K and Y' for approximate equality with a certain threshold.

So If I use maximumRelativeError for comparing the approximate equality for two matrices, then even though the difference is small for two cases, the relative error is significantly different. So there is a need for a new evaluation metric.

Goal:

The goal of this PR is to implement new evaluation metric meanAbsoluteError() between two matrices which takes the mean of all the absolute errors of individual elements.

Absolute error between a and b = abs(a-b).

So both the cases described above will have the same absolute error. So I propose this would be a good choice of metric for comparing two matrices for approximate equality as needed for testing optimizers.

[TMVA] Add new Evaluation Metric ( meanAbsoluteError between two matr…
…ices ) needed for evaluating the optimizers.

@ravikiran0606 ravikiran0606 requested a review from lmoneta as a code owner Jul 27, 2018

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 27, 2018

Can one of the admins verify this patch?

@ravikiran0606

This comment has been minimized.

Copy link
Contributor Author

commented Jul 27, 2018

@lmoneta @stwunsch Can you review this PR?

*/

//______________________________________________________________________________
template <typename Matrix1, typename Matrix2>

This comment has been minimized.

Copy link
@stwunsch

stwunsch Jul 27, 2018

Contributor

Is it needed to template the type of both matrices? Shouldn't they be of the same type?

This comment has been minimized.

Copy link
@stwunsch

stwunsch Jul 27, 2018

Contributor

I see that the implementation of the maximumRelativeError has done the some... Nevertheless, is it needed?

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 27, 2018

Author Contributor

Yeah, I guess we need the template arguments. Because in certain cases, the matrices can be of different types like TMatrixT or TCpuMatrix or TCudaMatrix.

This comment has been minimized.

Copy link
@stwunsch

stwunsch Jul 27, 2018

Contributor

alright, fine then.

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 27, 2018

@phsft-bot build

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 27, 2018

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dimt=ON -Dccache=ON
How to customize builds

@lmoneta

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2018

These changes are fine with me. I can merge it

@lmoneta lmoneta merged commit 78e3f0c into root-project:master Aug 2, 2018

2 checks passed

Jenkins CI build Build passed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@ravikiran0606 ravikiran0606 deleted the ravikiran0606:Evaluation-Metric branch Aug 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.