New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Neuro CMAES-Algorithm #938

Closed
wants to merge 108 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@kartik-nighania
Contributor

kartik-nighania commented Mar 14, 2017

I have used and modified Bang's genome and have implemeted cmaes into it ..
Tests are remaining for it ..

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Mar 14, 2017

Contributor

tests are remaining

Contributor

kartik-nighania commented Mar 14, 2017

tests are remaining

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Mar 15, 2017

Contributor

these test.. passed only with the build files right ?
test remains ..
@zoq

Contributor

kartik-nighania commented Mar 15, 2017

these test.. passed only with the build files right ?
test remains ..
@zoq

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Mar 15, 2017

Contributor

linkweights
@zoq here in the picture after every termination with lambda=24 population and then finding the mean and updating the search distribution some of the linkgene vector values at shown and mario keeps crashing with enemy at the start only..

Contributor

kartik-nighania commented Mar 15, 2017

linkweights
@zoq here in the picture after every termination with lambda=24 population and then finding the mean and updating the search distribution some of the linkgene vector values at shown and mario keeps crashing with enemy at the start only..

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Mar 17, 2017

Contributor

awesome...almost 200 lines will get removed. Thanks!!

@zoq now the fixing and testing will take at least 10 days and i am thinking of doing it from 1-10 April because
i also want to take part in this years GSOC. I deeply loved these neural evolution projects and so started working on it back then but this year as k_singh has already started working on it and is doing a great job .. I think i should start working on the RNN project. I'll still submit a proposal on reinforcement learning though.

how is your ann -> RNN code base..
I mean what should I implement before the final submission on RNN to make a strong proposal ?
Thank You

Contributor

kartik-nighania commented Mar 17, 2017

awesome...almost 200 lines will get removed. Thanks!!

@zoq now the fixing and testing will take at least 10 days and i am thinking of doing it from 1-10 April because
i also want to take part in this years GSOC. I deeply loved these neural evolution projects and so started working on it back then but this year as k_singh has already started working on it and is doing a great job .. I think i should start working on the RNN project. I'll still submit a proposal on reinforcement learning though.

how is your ann -> RNN code base..
I mean what should I implement before the final submission on RNN to make a strong proposal ?
Thank You

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Mar 18, 2017

Contributor

@zoq is the RNN code working.. :)

Contributor

kartik-nighania commented Mar 18, 2017

@zoq is the RNN code working.. :)

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Mar 22, 2017

Member

@kartik-nighania this looks really cool, I think instead of starting with the Mario task it would be a better idea to test it first on some easier tasks like CartPole. It might help to track down some hard to find errors. If you like I can prettify the tasks Bang has written last year and probably merge them. Perhaps it's helpful for some others too. What do you think?

how is your ann -> RNN code base..
I mean what should I implement before the final submission on RNN to make a strong proposal ?
is the RNN code working.. :)

About the RNN code, I have written various tasks for the RNN code take a look at: src/mlpack/tests/recurrent_network_test.cpp, I'd like to add more tests, but since we like to keep the test time somewhat low and the benefit is kinda limited, I haven't, maybe in the near future, I'll add some more tests.

I guess some recurrent network related addition could be the implementation of the GRU function or the Fast-LSTM function. But don't feel obligated.

Member

zoq commented Mar 22, 2017

@kartik-nighania this looks really cool, I think instead of starting with the Mario task it would be a better idea to test it first on some easier tasks like CartPole. It might help to track down some hard to find errors. If you like I can prettify the tasks Bang has written last year and probably merge them. Perhaps it's helpful for some others too. What do you think?

how is your ann -> RNN code base..
I mean what should I implement before the final submission on RNN to make a strong proposal ?
is the RNN code working.. :)

About the RNN code, I have written various tasks for the RNN code take a look at: src/mlpack/tests/recurrent_network_test.cpp, I'd like to add more tests, but since we like to keep the test time somewhat low and the benefit is kinda limited, I haven't, maybe in the near future, I'll add some more tests.

I guess some recurrent network related addition could be the implementation of the GRU function or the Fast-LSTM function. But don't feel obligated.

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Mar 22, 2017

Member

Also, maybe you can push the latest changes you have made (removing the eigenvalue code)?

Member

zoq commented Mar 22, 2017

Also, maybe you can push the latest changes you have made (removing the eigenvalue code)?

@mlpack-jenkins

This comment has been minimized.

Show comment
Hide comment
@mlpack-jenkins

mlpack-jenkins Apr 23, 2017

Can one of the admins verify this patch?

mlpack-jenkins commented Apr 23, 2017

Can one of the admins verify this patch?

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Apr 27, 2017

Contributor

the build is passing on my local machine. it is failing due to -
stochastic gradient descend test as below-
/home/travis/build/mlpack/mlpack/src/mlpack/tests/minibatch_sgd_test.cpp(50): fatal error in "SGDSimilarityTest": difference{0.0299955%} between sResult{-0.99970004499550047} and msResult{-0.99999999999999989} exceeds 1e-08%

Contributor

kartik-nighania commented Apr 27, 2017

the build is passing on my local machine. it is failing due to -
stochastic gradient descend test as below-
/home/travis/build/mlpack/mlpack/src/mlpack/tests/minibatch_sgd_test.cpp(50): fatal error in "SGDSimilarityTest": difference{0.0299955%} between sResult{-0.99970004499550047} and msResult{-0.99999999999999989} exceeds 1e-08%

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Apr 27, 2017

Contributor

@zoq i got a google identification rejection mail today. I re-uploaded my college ID card.
I think during this period my proposal would not have been available to the organisation.

gsoc

please consider my proposal too for the organisation.

Here is the google docs link if needed link

Contributor

kartik-nighania commented Apr 27, 2017

@zoq i got a google identification rejection mail today. I re-uploaded my college ID card.
I think during this period my proposal would not have been available to the organisation.

gsoc

please consider my proposal too for the organisation.

Here is the google docs link if needed link

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Apr 27, 2017

Member

@kartik-nighania Regarding the proof of enrolment please contact the GSoC support team, it sounds like your proof doesn't show the current semester. There is nothing we can do here from our/organization side.

Member

zoq commented Apr 27, 2017

@kartik-nighania Regarding the proof of enrolment please contact the GSoC support team, it sounds like your proof doesn't show the current semester. There is nothing we can do here from our/organization side.

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Apr 27, 2017

Member

Also, the proposal was available even if the proof of enrolment wasn't completed.

Member

zoq commented Apr 27, 2017

Also, the proposal was available even if the proof of enrolment wasn't completed.

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Apr 28, 2017

Contributor

@zoq thanks! it was a relief .. 👍 i i corrected and re-uploaded the POE..

Contributor

kartik-nighania commented Apr 28, 2017

@zoq thanks! it was a relief .. 👍 i i corrected and re-uploaded the POE..

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq May 9, 2017

Member

Shangtong has written some neat tasks (MountainCar and CartPole) that we can use to test different methods, so I think, we could write a simple test case for the CMAES algorithm instead using the mario task.

Here are the tasks: https://github.com/mlpack/mlpack/tree/master/src/mlpack/methods/reinforcement_learning/environment
and here is a simple test for the tasks: https://github.com/mlpack/mlpack/blob/master/src/mlpack/tests/rl_environment_test.cpp

Let me know what you think, I can also provide some more information if you like about how to use the code.

Member

zoq commented May 9, 2017

Shangtong has written some neat tasks (MountainCar and CartPole) that we can use to test different methods, so I think, we could write a simple test case for the CMAES algorithm instead using the mario task.

Here are the tasks: https://github.com/mlpack/mlpack/tree/master/src/mlpack/methods/reinforcement_learning/environment
and here is a simple test for the tasks: https://github.com/mlpack/mlpack/blob/master/src/mlpack/tests/rl_environment_test.cpp

Let me know what you think, I can also provide some more information if you like about how to use the code.

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania May 9, 2017

Contributor
Contributor

kartik-nighania commented May 9, 2017

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq May 9, 2017

Member

No problem, the actual coding phase hasn't started yet.

Member

zoq commented May 9, 2017

No problem, the actual coding phase hasn't started yet.

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania May 12, 2017

Contributor

Ill start with using the serialization code

Contributor

kartik-nighania commented May 12, 2017

Ill start with using the serialization code

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq May 12, 2017

Member

I think, instead of using Bang's code for the CMAS method, we could use the ann code. CMAS works on a fixed topology so the transition should be fairly easy. I can say more about what we have to do once we have a working example, e.g. by using the CartPole task. So, I would suggest to look into the CartPole first and to provide a simple example.

Member

zoq commented May 12, 2017

I think, instead of using Bang's code for the CMAS method, we could use the ann code. CMAS works on a fixed topology so the transition should be fairly easy. I can say more about what we have to do once we have a working example, e.g. by using the CartPole task. So, I would suggest to look into the CartPole first and to provide a simple example.

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania May 13, 2017

Contributor

@zoq the build passed :)

  1. zoq i changed bangs nets in cmaes to fixed topology only and doesnt takes any other data and is fairly simple. I am a bit scared to change to ann code as the final benchmark test im planning to do on the same neural net code (though different topology one will have a modified net code).
  2. the additional graphical net interface that i said as optional i think once made for one neat or cne will be easy to change to support other ones.
    i hope ur able to get what im trying to say.
    also i was thinking to start with the serialisation code first that can load and save the training data due to which further work will get easy. To implement on CNE and NEAT and to complete them first.. which will later benefit to make cmaes as ill be able to load and save and go for better debugging.

this is what i am planning for now. U decide what will be the best to do .. :)

Contributor

kartik-nighania commented May 13, 2017

@zoq the build passed :)

  1. zoq i changed bangs nets in cmaes to fixed topology only and doesnt takes any other data and is fairly simple. I am a bit scared to change to ann code as the final benchmark test im planning to do on the same neural net code (though different topology one will have a modified net code).
  2. the additional graphical net interface that i said as optional i think once made for one neat or cne will be easy to change to support other ones.
    i hope ur able to get what im trying to say.
    also i was thinking to start with the serialisation code first that can load and save the training data due to which further work will get easy. To implement on CNE and NEAT and to complete them first.. which will later benefit to make cmaes as ill be able to load and save and go for better debugging.

this is what i am planning for now. U decide what will be the best to do .. :)

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania May 13, 2017

Contributor

also the builds failed for Bang which i want to correct, modify according to this years proposal and merge first ..

Contributor

kartik-nighania commented May 13, 2017

also the builds failed for Bang which i want to correct, modify according to this years proposal and merge first ..

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq May 14, 2017

Member

I get your point, I think the transition to the ann codebase should be fairly simple and straightforward, I can help you with that, once we have a test case. About the API differences, we should be able and should also try to provide the same or mostly the same interface independent from the codebase. Another benefit of using the ann code is that we are able to optimize e.g. a Logistic-Regression or Linear-Regression model with the CMAES method, just because the ann code shares the same interface. So, if you like the idea, I would continue with a simple test case for the current CMAES method and afterwards, I can give you pointers what we have to change to make it work with the ann code

Member

zoq commented May 14, 2017

I get your point, I think the transition to the ann codebase should be fairly simple and straightforward, I can help you with that, once we have a test case. About the API differences, we should be able and should also try to provide the same or mostly the same interface independent from the codebase. Another benefit of using the ann code is that we are able to optimize e.g. a Logistic-Regression or Linear-Regression model with the CMAES method, just because the ann code shares the same interface. So, if you like the idea, I would continue with a simple test case for the current CMAES method and afterwards, I can give you pointers what we have to change to make it work with the ann code

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania May 16, 2017

Contributor

ohkae @zoq . after u write a test case on ann ill start to pick from there.. Below is the implementation on cmaes. Im also in a doubt of its working and will try to put it on small tasks first..

https://github.com/kartik-nighania/CMA-ES_SuperMario/blob/master/SuperMarioBros/super_mario_bros.cpp

Contributor

kartik-nighania commented May 16, 2017

ohkae @zoq . after u write a test case on ann ill start to pick from there.. Below is the implementation on cmaes. Im also in a doubt of its working and will try to put it on small tasks first..

https://github.com/kartik-nighania/CMA-ES_SuperMario/blob/master/SuperMarioBros/super_mario_bros.cpp

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq May 18, 2017

Member

Hello, here: https://github.com/zoq/mlpack/tree/cmaes is a basic concept, I removed almost all CMA-ES functions and just show how the ann code can be used in conjunction with a potential CMAES optimizer class. The important files are cmaes.hpp, cmaes_impl.hpp in methods/cmaes and the cmaes_test.cpp file in tests. Also, the code is based on the existing optimizer classes (core/optimizer).

Each function that can be trained with the CMA-ES method should expose at least:

  • NumFunctions() - Returns the number of trainable functions e.g. number of env's.
  • Evaluate(...) - Evaluates function x with the current parameter.
  • Parameters() - Trainable parameters.

That way we can not only train on some env's we could also use the method to optimize e.g. logistic regression.

Let me know what you think, and don't hesitate to ask questions if anything isn't clear, as I said this is just a basic concept.

Member

zoq commented May 18, 2017

Hello, here: https://github.com/zoq/mlpack/tree/cmaes is a basic concept, I removed almost all CMA-ES functions and just show how the ann code can be used in conjunction with a potential CMAES optimizer class. The important files are cmaes.hpp, cmaes_impl.hpp in methods/cmaes and the cmaes_test.cpp file in tests. Also, the code is based on the existing optimizer classes (core/optimizer).

Each function that can be trained with the CMA-ES method should expose at least:

  • NumFunctions() - Returns the number of trainable functions e.g. number of env's.
  • Evaluate(...) - Evaluates function x with the current parameter.
  • Parameters() - Trainable parameters.

That way we can not only train on some env's we could also use the method to optimize e.g. logistic regression.

Let me know what you think, and don't hesitate to ask questions if anything isn't clear, as I said this is just a basic concept.

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania May 19, 2017

Contributor

Thanks @zoq ill look to it

Contributor

kartik-nighania commented May 19, 2017

Thanks @zoq ill look to it

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania May 19, 2017

Contributor

Now i can get where you are going.. to make them more than just some game algos..

Contributor

kartik-nighania commented May 19, 2017

Now i can get where you are going.. to make them more than just some game algos..

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq May 20, 2017

Member

Right, since CMA-ES does not alter the topology of the method, the underlying parameter structure is completely independent of the problem itself. So We could optimize some neural network or as I said a logistic regression model. As long as the method implements a method that returns the parameter, we can easily swap the optimization process. I think that's pretty neat and I haven't seen something like that in another library.

Member

zoq commented May 20, 2017

Right, since CMA-ES does not alter the topology of the method, the underlying parameter structure is completely independent of the problem itself. So We could optimize some neural network or as I said a logistic regression model. As long as the method implements a method that returns the parameter, we can easily swap the optimization process. I think that's pretty neat and I haven't seen something like that in another library.

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Aug 6, 2017

Contributor

@zoq
when i open the html file created by doxygen.
i cant see the latex functions

\f
f(x,y,z) = -e^{-|x|} + y^2 + z^4 + 3*z^2
\f

created properly.
will this and other function get fixed ?

Contributor

kartik-nighania commented Aug 6, 2017

@zoq
when i open the html file created by doxygen.
i cant see the latex functions

\f
f(x,y,z) = -e^{-|x|} + y^2 + z^4 + 3*z^2
\f

created properly.
will this and other function get fixed ?

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Aug 6, 2017

Member

Have to take a look at the function issue, thanks for the information.

Member

zoq commented Aug 6, 2017

Have to take a look at the function issue, thanks for the information.

@zoq

Looks good, just made a minor comment; there are some style issue that we should fix later, the same issue are in the CMAES code.

Show outdated Hide outdated doc/tutorials/cmaes/cmaes.txt
@zoq

Just made a bunch of pedantic style comments.

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Aug 9, 2017

Contributor

// modifying the parameters of the vanilla network
function.parameters() = parameters; // parameter is a column vector.

is this the way todo it ? as Evaluate function takes the network parameters of vanilla network but doesnt uses it ?

also @zoq how to open a new PR for CNE ?

the temination criteria set by Bang for task as the function is the having a different parameter. for using it as an optimizer i think -

  1. max number of iterations
  2. minimum function value history change since last generation just like tolerence in SGD
  3. or maximum number of epoch

maximum number of epoch and the above parameter line makes it specific to neural network only. What to do here?

ill push the above suggested changes by today late night..

Contributor

kartik-nighania commented Aug 9, 2017

// modifying the parameters of the vanilla network
function.parameters() = parameters; // parameter is a column vector.

is this the way todo it ? as Evaluate function takes the network parameters of vanilla network but doesnt uses it ?

also @zoq how to open a new PR for CNE ?

the temination criteria set by Bang for task as the function is the having a different parameter. for using it as an optimizer i think -

  1. max number of iterations
  2. minimum function value history change since last generation just like tolerence in SGD
  3. or maximum number of epoch

maximum number of epoch and the above parameter line makes it specific to neural network only. What to do here?

ill push the above suggested changes by today late night..

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Aug 9, 2017

Member

// modifying the parameters of the vanilla network
function.parameters() = parameters; // parameter is a column vector.

is this the way todo it ? as Evaluate function takes the network parameters of vanilla network but doesnt uses it ?

The correct way would be to update the parameters passed to the Optimize function e.g. iterate which is a reference to the network parameter.

iterate = parameters;

Also, you are right, the parameters in the evaluation function aren't used because we expect that the optimizer uses the reference, so that we don't have to reset the weights for each call of Evaluate, which could be slow.

also @zoq how to open a new PR for CNE ?

I would recommend to create a new branch: https://www.atlassian.com/git/tutorials/using-branches.

the temination criteria set by Bang for task as the function is the having a different parameter. for using it as an optimizer i think -

  1. max number of iterations
  2. minimum function value history change since last generation just like tolerence in SGD
  3. or maximum number of epoch
  4. maximum number of epoch and the above parameter line makes it specific to neural network only. What to do here?

I've checked the code and I'm not sure what you mean with maximum number of epoch, can you point me to the code/line? As you already pointed out 1. is the equivalent of number of iterations (SGD) and minimum function value is the tolerance parameter (SGD).

Member

zoq commented Aug 9, 2017

// modifying the parameters of the vanilla network
function.parameters() = parameters; // parameter is a column vector.

is this the way todo it ? as Evaluate function takes the network parameters of vanilla network but doesnt uses it ?

The correct way would be to update the parameters passed to the Optimize function e.g. iterate which is a reference to the network parameter.

iterate = parameters;

Also, you are right, the parameters in the evaluation function aren't used because we expect that the optimizer uses the reference, so that we don't have to reset the weights for each call of Evaluate, which could be slow.

also @zoq how to open a new PR for CNE ?

I would recommend to create a new branch: https://www.atlassian.com/git/tutorials/using-branches.

the temination criteria set by Bang for task as the function is the having a different parameter. for using it as an optimizer i think -

  1. max number of iterations
  2. minimum function value history change since last generation just like tolerence in SGD
  3. or maximum number of epoch
  4. maximum number of epoch and the above parameter line makes it specific to neural network only. What to do here?

I've checked the code and I'm not sure what you mean with maximum number of epoch, can you point me to the code/line? As you already pointed out 1. is the equivalent of number of iterations (SGD) and minimum function value is the tolerance parameter (SGD).

kartik-nighania added some commits Aug 10, 2017

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Aug 19, 2017

Contributor

@zoq
I think this PR is also ready.

perfect rosenbrock is passing now.
and -nan is fixed totally.
for cross check i ran the program 1100 times also.

Contributor

kartik-nighania commented Aug 19, 2017

@zoq
I think this PR is also ready.

perfect rosenbrock is passing now.
and -nan is fixed totally.
for cross check i ran the program 1100 times also.

kartik-nighania added some commits Aug 19, 2017

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Aug 28, 2017

Member

I'd like to take a closer look into the eigenvalue calculation, before merging this in (in the next days (2-3)). Also, besides some minor style issues, which I can solve once this is merged, I think this is ready.

Member

zoq commented Aug 28, 2017

I'd like to take a closer look into the eigenvalue calculation, before merging this in (in the next days (2-3)). Also, besides some minor style issues, which I can solve once this is merged, I think this is ready.

@kartik-nighania

This comment has been minimized.

Show comment
Hide comment
@kartik-nighania

kartik-nighania Aug 28, 2017

Contributor

@zoq
thanks

currently in process of writing final report and summary. :)

Contributor

kartik-nighania commented Aug 28, 2017

@zoq
thanks

currently in process of writing final report and summary. :)

@sourabhvarshney111

This comment has been minimized.

Show comment
Hide comment
@sourabhvarshney111

sourabhvarshney111 Mar 17, 2018

Contributor

@kartik-nighania I think work of this PR is completed. @zoq Can this PR be merged?

Contributor

sourabhvarshney111 commented Mar 17, 2018

@kartik-nighania I think work of this PR is completed. @zoq Can this PR be merged?

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Apr 2, 2018

Member

You are right, thanks @kartik-nighania for the work.

Member

zoq commented Apr 2, 2018

You are right, thanks @kartik-nighania for the work.

@zoq zoq closed this Apr 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment