Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TMVA] API-Support for SGD Optimizer: #2309

Merged
merged 1 commit into from Jul 19, 2018

Conversation

5 participants
@ravikiran0606
Copy link
Contributor

commented Jul 10, 2018

[TMVA] API-Support for SGD Optimizer:

  • Add Base Class VOptimizer.
  • Add Derived Class TSGD with Momentum implementation.
  • Add Unit Tests for SGD Optimizer.
  • Modify the MethodDL Tests to include parsing options for Optimizer.

An example Training Strategy string may look like,

"LearningRate=1e-1,Optimizer=SGD,Momentum=0.9,Repetitions=1,"
"ConvergenceSteps=20,BatchSize=256,TestRepetitions=10,"
"WeightDecay=1e-4,Regularization=L2,"
"DropConfig=0.0+0.5+0.5+0.5, Multithreading=True"

Reference Implementation: Tensorflow
Blog Post: https://www.sravikiran.com/GSOC18//2018/07/09/sgd/

@ravikiran0606 ravikiran0606 requested review from amadio and lmoneta as code owners Jul 10, 2018

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 10, 2018

Can one of the admins verify this patch?

@ravikiran0606 ravikiran0606 changed the title SGD Optimizer API Support for SGD Optimizer Jul 10, 2018

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2018

@phsft-bot build

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 10, 2018

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dimt=ON -Dccache=ON
How to customize builds

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 10, 2018

Build failed on centos7/clang39.
See console output.

Warnings:

  • ../root/core/metacling/src/TClingValue.cxx:51:25: warning: comparison of integers of different signs: 'int' and 'size_type' (aka 'unsigned long') [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:92:23: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:95:26: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:105:26: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:92:23: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:95:26: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:105:26: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:92:23: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:95:26: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]
  • include/TMVA/DNN/SGD.h:105:26: warning: comparison of integers of different signs: 'int' and 'unsigned long' [-Wsign-compare]

And 3 more

Failing tests:

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2018

Can one of the admins verify this patch?

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2018

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dimt=ON -Dccache=ON
How to customize builds

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2018

you should see the warnings regarding the comparisons of unsigned long and int as well on your system. can you check? the tests for all builds seem to be fine. is it possible for you to access cdash.cern.ch?

as well, please rebase and squash your commits in a single commit with a meaningful message.

last change: please put a [TMVA] tag at the begin of your commit message so that it's clear what you've worked on in the commit history.

@ravikiran0606 ravikiran0606 force-pushed the ravikiran0606:SGD-Optimizer branch from 626d23f to 779de01 Jul 11, 2018

@ravikiran0606

This comment has been minimized.

Copy link
Contributor Author

commented Jul 11, 2018

@phsft-bot build

@ravikiran0606 ravikiran0606 changed the title API Support for SGD Optimizer [TMVA] API-Support for SGD Optimizer: Jul 11, 2018

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2018

you cannot trigger the PR builds ;) have you fixed the warnings due to the type comparisons?

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2018

@phsft-bot build

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2018

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dimt=ON -Dccache=ON
How to customize builds

@ravikiran0606

This comment has been minimized.

Copy link
Contributor Author

commented Jul 11, 2018

@stwunsch Yes, I have fixed the warnings with type comparisons and squashed the commits into a single commit.

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2018

Build failed on centos7/gcc62.
See console output.

Warnings:

  • ../root/core/metacling/src/TClingValue.cxx:51:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]

Failing tests:

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2018

Build failed on centos7/gcc7.
See console output.

Warnings:

  • ../root/core/metacling/src/TClingValue.cxx:51:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  • ../root/core/newdelete/src/NewDelete.cxx:217:11: warning: ‘operator new’ must not return NULL unless it is declared ‘throw()’ (or -fcheck-new is in effect)
  • ../root/core/newdelete/src/NewDelete.cxx:344:11: warning: ‘operator new’ must not return NULL unless it is declared ‘throw()’ (or -fcheck-new is in effect)
  • ../root/core/newdelete/src/NewDelete.cxx:263:6: warning: the program should also define ‘void operator delete(void*, std::size_t)’ [-Wsized-deallocation]
  • ../root/core/newdelete/src/NewDelete.cxx:367:6: warning: the program should also define ‘void operator delete [](void*, std::size_t)’ [-Wsized-deallocation]

Failing tests:

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2018

Build failed on slc6/gcc48.
See console output.

Warnings:

  • ../root/core/metacling/src/TClingValue.cxx:51:39: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]

Failing tests:

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 12, 2018

@lmoneta i think that we are good to merge. can you check?

@@ -76,6 +76,11 @@ enum class EInitialization {
kGlorotUniform = 'F',
};

/* Enum representing the optimizer used for training. */

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

Can you use Doxygen style comments: /// Enum representing the optimizer used for training.

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

I used this way of commenting to ensure consistency since other enums in the same file were using this style for commenting. So should I need to change this?

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

Hmm bad and totally not your fault! Please use a style that at least doxygen-documents yours.

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 15, 2018

Author Contributor

I changed this.

inline void IncrementGlobalStep() { this->fGlobalStep++; }

/*! Getters */
inline Scalar_t GetLearningRate() { return fLearningRate; }

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

Several of these should be const!

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

Fixed.

void Step();

/*! Virtual Destructor. */
virtual ~VOptimizer();

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

Please use virtual ~VOptimizer() = default;

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

Fixed.

typename DeepNet_t = TDeepNet<Architecture_t, Layer_t>>
class VOptimizer {
using Matrix_t = typename Architecture_t::Matrix_t;
using Scalar_t = typename Architecture_t::Scalar_t;

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

These are private using declarations, but they are used by public interfaces. Could you make these types public, please?

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

Fixed.

/*! Getters */
inline Scalar_t GetLearningRate() { return fLearningRate; }
inline size_t GetGlobalStep() { return fGlobalStep; }
inline std::vector<Layer_t *> GetLayers() { return fDeepNet.GetLayers(); }

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

Do you really want to copy the vector?

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

hmm! I changed that.

break;
default:
optimizer = std::unique_ptr<DNN::TSGD<Architecture_t, Layer_t, DeepNet_t>>(
new DNN::TSGD<Architecture_t, Layer_t, DeepNet_t>(settings.learningRate, deepNet, settings.momentum));

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

As this case is identical with the previous, please combine them, adding a commend /*Intentional fall-through*/

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

So Shall I remove the default case and have only one case for EOptimizer::kSGD for now?

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

I don't care; maybe @lmoneta has an opinion. But having a second, default case for a single-valued enum is - surprising.

// copy configuration when reached a minimum error
if (testError < minTestError ) {
// Copy weights from deepNet to fNet
Log() << std::setw(10) << stepCount << " Minimun Test error found - save the configuration " << Endl;
Log() << std::setw(10) << optimizer->GetGlobalStep()

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

Note that from here on, every one else writing to Log() (which defaults to std::cerr or std::cout?) will have "suffer" from the setw(10) call. Please consider capturing the flags and resetting them afterwards. See e.g. https://stackoverflow.com/a/2273352/6182509

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

It was like this previously. I haven't changed that part of the code. And I am not able to get what change I want to do here?

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

OK fine - we'll fix it when people complain!

* (http://tmva.sourceforge.net/LICENSE) *
**********************************************************************************/

#ifndef TMVA_TEST_DNN_TEST_OPTIMIZER_H

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

Can you match the file name, i.e. ...TEST_OPTIMIZATION_H (not OPTIMIZER)

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

Yeah, Fixed.

if (error > 1e-5) {
return 1;
}
}

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

Please add a trailing newline.

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

Mmm! Fixed.

std::cout << "Stochastic Gradient Descent with momentum: Maximum relative error = " << error << std::endl;
if (error > 1e-5) {
return 1;
}

This comment has been minimized.

Copy link
@Axel-Naumann

Axel-Naumann Jul 12, 2018

Member

You're missing a crucial return 0! This shouldn't succeed without... Did you run this test and it worked?

This comment has been minimized.

Copy link
@ravikiran0606

ravikiran0606 Jul 12, 2018

Author Contributor

Yeah! The test worked. Anyways I'll add this.

@ravikiran0606 ravikiran0606 force-pushed the ravikiran0606:SGD-Optimizer branch from 779de01 to 2af6499 Jul 15, 2018

@ravikiran0606

This comment has been minimized.

Copy link
Contributor Author

commented Jul 15, 2018

@stwunsch @lmoneta I have made changes as said by @Axel-Naumann. Can you please review them?

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 16, 2018

@phsft-bot build

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 16, 2018

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dimt=ON -Dccache=ON
How to customize builds

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 16, 2018

@ravikiran0606 clang-format does not seem to be happy with the formatting of the file headers. can you check?

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 16, 2018

PR builds without errors or warnings, nice!

@ravikiran0606 ravikiran0606 force-pushed the ravikiran0606:SGD-Optimizer branch from 2af6499 to db0048c Jul 16, 2018

[TMVA] API-Support for SGD Optimizer:
* Implement Base Class VOptimizer

* Implement Class TSGD with Momentum method

* Add Unit Tests for TSGD

* Add Parsing Options for Optimizers

* Modify MethodDL to include the optimizer TSGD

* Add ROOT Style docs and clang-format the code

@ravikiran0606 ravikiran0606 force-pushed the ravikiran0606:SGD-Optimizer branch from db0048c to 49ba800 Jul 16, 2018

@stwunsch

This comment has been minimized.

Copy link
Contributor

commented Jul 16, 2018

@phsft-bot build

@phsft-bot

This comment has been minimized.

Copy link
Collaborator

commented Jul 16, 2018

Starting build on slc6/gcc48, slc6-i686/gcc49, centos7/clang39, centos7/gcc62, centos7/gcc7, fedora28/native, ubuntu16/native, mac1013/native, windows10/vc15 with flags -Dimt=ON -Dccache=ON
How to customize builds

@lmoneta
Copy link
Contributor

left a comment

Why the Optimizer class needs to be templated on the Layer type and the network ?
Would not be enough to be templated only on the Architecture ?

@ravikiran0606

This comment has been minimized.

Copy link
Contributor Author

commented Jul 19, 2018

@stwunsch @lmoneta As we discussed, the three template arguments are needed because I use them inside the class definition of the VOptimizer class. I actually thought of doing so by getting ideas from the DeepNet class, which has two template arguments, one for Architecture and another for Layer. So I did use a similar approach.

@ravikiran0606

This comment has been minimized.

Copy link
Contributor Author

commented Jul 19, 2018

@stwunsch I have also updated the PR with the clang formatting. Now the Travis ci build succeds with clang format option too. I guess now we are good to merge! :D

@stwunsch stwunsch merged commit 5bf9b81 into root-project:master Jul 19, 2018

2 checks passed

Jenkins CI build Build passed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@ravikiran0606 ravikiran0606 deleted the ravikiran0606:SGD-Optimizer branch Jul 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.