New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Genetic optimization #190

Open
chrisdk2015 opened this Issue Oct 23, 2015 · 7 comments

Comments

Projects
None yet
5 participants
@chrisdk2015

chrisdk2015 commented Oct 23, 2015

Created this issue for tracking the genetic optimization

https://github.com/chrisdk2015/LeanOptimization

It uses the GAF framework and is currently for running on the command line.

Howto parallize this would be nice to know. Maybe one needs to change the GAF framework.

@bizcad

This comment has been minimized.

Show comment
Hide comment
@bizcad

bizcad Oct 27, 2015

Contributor

@chrisdk2015

I got the genetic algorithm running without blurring by on the console. However, after 1723 trials, it blows up my Visual Studio. I mean VS just quits with no notice or anything. It has happened twice, once with console blur and once without. It seems to happen in the same place but I cannot catch an Exception.

I was wondering how many trials were needed to resolve the genetic algorithm. And also did your crash like that. Of course, if you were waiting 40 seconds between trials, it would have taken all night. My current version did the 1723 trials in 1:09:22 or 2-3 seconds per trial and only shows the 1 line per trial in the console.

I copied ConsoleResultsHandler and modified it into OptimizationResutsHandler. I also commented out line 62 in JobQueue because it was Log.Trace("dodah") before the Log.Handler was set to a FileLogHandler.

Here is a link to a dropbox folder with the log.txt and log.csv files for the 1723 trials.
https://www.dropbox.com/sh/zm19bgmuo1t5os0/AACt4pHp6zbE1KNmWxnhlO7wa?dl=0

I am pushing the complete project to https://github.com/bizcad/LeanOpto

On line 89 of Program, I set

            Log.LogHandler = (ILogHandler)new FileLogHandler();
            Log.DebuggingEnabled = true;
            Log.DebuggingLevel = 2;

so that Log output is only sent to a file. Then I replace all the Console.WriteLines (except for 2) to Log.Trace, Debug or Error.

I also commented the parts that refer to writing chart files. We do not need them.

Nick

Contributor

bizcad commented Oct 27, 2015

@chrisdk2015

I got the genetic algorithm running without blurring by on the console. However, after 1723 trials, it blows up my Visual Studio. I mean VS just quits with no notice or anything. It has happened twice, once with console blur and once without. It seems to happen in the same place but I cannot catch an Exception.

I was wondering how many trials were needed to resolve the genetic algorithm. And also did your crash like that. Of course, if you were waiting 40 seconds between trials, it would have taken all night. My current version did the 1723 trials in 1:09:22 or 2-3 seconds per trial and only shows the 1 line per trial in the console.

I copied ConsoleResultsHandler and modified it into OptimizationResutsHandler. I also commented out line 62 in JobQueue because it was Log.Trace("dodah") before the Log.Handler was set to a FileLogHandler.

Here is a link to a dropbox folder with the log.txt and log.csv files for the 1723 trials.
https://www.dropbox.com/sh/zm19bgmuo1t5os0/AACt4pHp6zbE1KNmWxnhlO7wa?dl=0

I am pushing the complete project to https://github.com/bizcad/LeanOpto

On line 89 of Program, I set

            Log.LogHandler = (ILogHandler)new FileLogHandler();
            Log.DebuggingEnabled = true;
            Log.DebuggingLevel = 2;

so that Log output is only sent to a file. Then I replace all the Console.WriteLines (except for 2) to Log.Trace, Debug or Error.

I also commented the parts that refer to writing chart files. We do not need them.

Nick

@chrisdk2015

This comment has been minimized.

Show comment
Hide comment
@chrisdk2015

chrisdk2015 Oct 27, 2015

@bizcad

I think there is a memory leak somewhere.

At least here when I run start running it the memory usage keeps going up.

Maybe you can put the memory usage in the log file to see if that is the reason.

I tried to do a profiling here with mono but I just get segfaults so it is of no help. Maybe you could profile it in VS? I think the profile there is better.

With the OptimizationResultHandler it goes a bit faster.

For the termination criteria you could use the following

private static double oldFitness = Double.NaN;

public static bool Terminate(Population population, 
            int currentGeneration, long currentEvaluation)
        {
            var fittest = population.GetTop(1)[0];
            if (oldFitness != Double.NaN) {
                var diff = System.Math.Abs (fittest.Fitness - oldFitness);
                if (diff < 1E-02)
                    return true;
            }
            oldFitness = fittest.Fitness;
            return false;
            //return currentGeneration > 10;
        }

instead of using the generation number to stop.

Also to avoid running that many times the number of chromosomes and genes per chromosome could be lowered

//create the chromosomes
            for (var p = 0; p < 10; p++)
            {

                var chromosome = new Chromosome();
                for (int i = 0; i < 10; i++) {
                    ConfigVars v = new ConfigVars ();
                    v.vars ["EMA_VAR1"] = RandomNumberBetweenInt (0, 20);
                    v.vars ["EMA_VAR2"] = RandomNumberBetweenInt (0, 100);

                    chromosome.Genes.Add (new Gene (v));
                }
                chromosome.Genes.ShuffleFast();
                population.Solutions.Add(chromosome);
            }

The question is how big the default population should be? I don't know.

By the way the code in my github has a solution for multiple variables using the ConfigVars class.

chrisdk2015 commented Oct 27, 2015

@bizcad

I think there is a memory leak somewhere.

At least here when I run start running it the memory usage keeps going up.

Maybe you can put the memory usage in the log file to see if that is the reason.

I tried to do a profiling here with mono but I just get segfaults so it is of no help. Maybe you could profile it in VS? I think the profile there is better.

With the OptimizationResultHandler it goes a bit faster.

For the termination criteria you could use the following

private static double oldFitness = Double.NaN;

public static bool Terminate(Population population, 
            int currentGeneration, long currentEvaluation)
        {
            var fittest = population.GetTop(1)[0];
            if (oldFitness != Double.NaN) {
                var diff = System.Math.Abs (fittest.Fitness - oldFitness);
                if (diff < 1E-02)
                    return true;
            }
            oldFitness = fittest.Fitness;
            return false;
            //return currentGeneration > 10;
        }

instead of using the generation number to stop.

Also to avoid running that many times the number of chromosomes and genes per chromosome could be lowered

//create the chromosomes
            for (var p = 0; p < 10; p++)
            {

                var chromosome = new Chromosome();
                for (int i = 0; i < 10; i++) {
                    ConfigVars v = new ConfigVars ();
                    v.vars ["EMA_VAR1"] = RandomNumberBetweenInt (0, 20);
                    v.vars ["EMA_VAR2"] = RandomNumberBetweenInt (0, 100);

                    chromosome.Genes.Add (new Gene (v));
                }
                chromosome.Genes.ShuffleFast();
                population.Solutions.Add(chromosome);
            }

The question is how big the default population should be? I don't know.

By the way the code in my github has a solution for multiple variables using the ConfigVars class.

@bizcad

This comment has been minimized.

Show comment
Hide comment
@bizcad

bizcad Oct 27, 2015

Contributor

@chrisdk2015 I am not experienced with the Microsoft profiler, but I tried running it. The data file filled up what was left of my C:\ drive. :o( I deleted the file and ran again stopping after about 200 iterations. The resulting report did not tell me anything useful. I dumped the profiler and ran the Program.cs.

I watched the Task manager for memory usage and it did not grow for 2.5 hours. So I do not believe there is a memory leak.

I put in a stop at interation 1721 to try to catch the error, but I formed the message incorrectly: string.Format("{1}", iterationnumber); A really stupid mistake. I changed the {1} to {0}. Now I need to run it again to get to the stop point. It is taking about 2.5 seconds per iteration so it is a 2.5 hour wait.

After I get there I will try integrating your new code. It will probably be tomorrow before I have something.

Nick

Contributor

bizcad commented Oct 27, 2015

@chrisdk2015 I am not experienced with the Microsoft profiler, but I tried running it. The data file filled up what was left of my C:\ drive. :o( I deleted the file and ran again stopping after about 200 iterations. The resulting report did not tell me anything useful. I dumped the profiler and ran the Program.cs.

I watched the Task manager for memory usage and it did not grow for 2.5 hours. So I do not believe there is a memory leak.

I put in a stop at interation 1721 to try to catch the error, but I formed the message incorrectly: string.Format("{1}", iterationnumber); A really stupid mistake. I changed the {1} to {0}. Now I need to run it again to get to the stop point. It is taking about 2.5 seconds per iteration so it is a 2.5 hour wait.

After I get there I will try integrating your new code. It will probably be tomorrow before I have something.

Nick

@gururise

This comment has been minimized.

Show comment
Hide comment
@gururise

gururise Nov 3, 2015

Been trying to get this running the past two days, I keep crashing on line 356 in Program.cs:
var res = (double)rc.Run (val);
On the terminal, I get this error:

20151103 08:52:34 ERROR:: Loader.TryCreateILAlgorithm(2): Sequence contains no matching element
20151103 08:52:34 Trace:: Initializing...
20151103 08:52:34 ERROR:: Engine.Main(): Error running algorithm: Object reference not set to an instance of an object >> at QuantConnect.Lean.Engine.Results.ConsoleResultHandler.RuntimeError (System.String message, System.String stacktrace) [0x0000f] in /home/gene/Downloads/Lean/Engine/Results/ConsoleResultHandler.cs:248
at QuantConnect.Lean.Engine.Engine.Run (QuantConnect.Packets.AlgorithmNodePacket job, System.String assemblyPath) [0x00305] in /home/gene/Downloads/Lean/Engine/Engine.cs:249

In my IDE (mono-develop), I get:

System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.

gururise commented Nov 3, 2015

Been trying to get this running the past two days, I keep crashing on line 356 in Program.cs:
var res = (double)rc.Run (val);
On the terminal, I get this error:

20151103 08:52:34 ERROR:: Loader.TryCreateILAlgorithm(2): Sequence contains no matching element
20151103 08:52:34 Trace:: Initializing...
20151103 08:52:34 ERROR:: Engine.Main(): Error running algorithm: Object reference not set to an instance of an object >> at QuantConnect.Lean.Engine.Results.ConsoleResultHandler.RuntimeError (System.String message, System.String stacktrace) [0x0000f] in /home/gene/Downloads/Lean/Engine/Results/ConsoleResultHandler.cs:248
at QuantConnect.Lean.Engine.Engine.Run (QuantConnect.Packets.AlgorithmNodePacket job, System.String assemblyPath) [0x00305] in /home/gene/Downloads/Lean/Engine/Engine.cs:249

In my IDE (mono-develop), I get:

System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.

@mchandschuh

This comment has been minimized.

Show comment
Hide comment
@mchandschuh

mchandschuh Nov 3, 2015

Member

20151103 08:52:34 ERROR:: Loader.TryCreateILAlgorithm(2): Sequence contains
no matching element

This message sounds like the Loader was unable to find the algorithm
specified in configuration.

The function _multipleTypeNameResolverFunction gets invoked with all type
names resolved from the assembly to select a single algorithm type to
instantiate. In many places in the code, the Loader is initialized to force
exactly one match on the type name via the LINQ .Single() call.

Check config.json 'algorithm-type-name' matches properly. Also, try
stepping through the invocation of the '_multipleTypeNameResolverFunction'
function from within the TryCreateILAlgorithm method in the Loader class.

-Michael

On Tue, Nov 3, 2015 at 11:59 AM, gururise notifications@github.com wrote:

Been trying to get this running the past two days, I keep crashing on line
356 in Program.cs:
var res = (double)rc.Run (val);
On the terminal, I get this error:

20151103 08:52:34 ERROR:: Loader.TryCreateILAlgorithm(2): Sequence
contains no matching element
20151103 08:52:34 Trace:: Initializing...
20151103 08:52:34 ERROR:: Engine.Main(): Error running algorithm: Object
reference not set to an instance of an object >> at
QuantConnect.Lean.Engine.Results.ConsoleResultHandler.RuntimeError
(System.String message, System.String stacktrace) [0x0000f] in
/home/gene/Downloads/Lean/Engine/Results/ConsoleResultHandler.cs:248
at QuantConnect.Lean.Engine.Engine.Run
(QuantConnect.Packets.AlgorithmNodePacket job, System.String assemblyPath)
[0x00305] in /home/gene/Downloads/Lean/Engine/Engine.cs:249

In my IDE (mono-develop), I get:

System.Collections.Generic.KeyNotFoundException: The given key was not
present in the dictionary.


Reply to this email directly or view it on GitHub
#190 (comment).

Member

mchandschuh commented Nov 3, 2015

20151103 08:52:34 ERROR:: Loader.TryCreateILAlgorithm(2): Sequence contains
no matching element

This message sounds like the Loader was unable to find the algorithm
specified in configuration.

The function _multipleTypeNameResolverFunction gets invoked with all type
names resolved from the assembly to select a single algorithm type to
instantiate. In many places in the code, the Loader is initialized to force
exactly one match on the type name via the LINQ .Single() call.

Check config.json 'algorithm-type-name' matches properly. Also, try
stepping through the invocation of the '_multipleTypeNameResolverFunction'
function from within the TryCreateILAlgorithm method in the Loader class.

-Michael

On Tue, Nov 3, 2015 at 11:59 AM, gururise notifications@github.com wrote:

Been trying to get this running the past two days, I keep crashing on line
356 in Program.cs:
var res = (double)rc.Run (val);
On the terminal, I get this error:

20151103 08:52:34 ERROR:: Loader.TryCreateILAlgorithm(2): Sequence
contains no matching element
20151103 08:52:34 Trace:: Initializing...
20151103 08:52:34 ERROR:: Engine.Main(): Error running algorithm: Object
reference not set to an instance of an object >> at
QuantConnect.Lean.Engine.Results.ConsoleResultHandler.RuntimeError
(System.String message, System.String stacktrace) [0x0000f] in
/home/gene/Downloads/Lean/Engine/Results/ConsoleResultHandler.cs:248
at QuantConnect.Lean.Engine.Engine.Run
(QuantConnect.Packets.AlgorithmNodePacket job, System.String assemblyPath)
[0x00305] in /home/gene/Downloads/Lean/Engine/Engine.cs:249

In my IDE (mono-develop), I get:

System.Collections.Generic.KeyNotFoundException: The given key was not
present in the dictionary.


Reply to this email directly or view it on GitHub
#190 (comment).

@bizcad

This comment has been minimized.

Show comment
Hide comment
@bizcad

bizcad Nov 3, 2015

Contributor

@gururise If you copied the code from @chrisdk2015 Optimization project on github, my guess is your problem is on line 154 of Program.cs

154 _resultshandler = new DesktopResultHandler ();

Try changing all the DesktopResultHandler to ConsoleResultHandler. DesktopResultHandler expects a Desktop application to write results to. In the console application there is no desktop.

Also take a look at my take on a complete solution including Lean and all it's sub-projects. This link is also in my reply above. I changed DesktopResultHandler in three places to an OptimizationResultHandler with some commented lines next for ConsoleResultHandler. If you want to see the results flash by in the console comment out the OptimizationResultsHandler and uncomment the ConsoleResultsHandler.

The code for the OptimizationResultsHandler is in the Engine\Results folder and is based upon ConsoleResultsHandler.

I run in Windows/VisualStudio, so I place a breakpoint about line 75 of Program.cs at the top of the LaunchLean function and step from there to find my error.

            Config.Set("environment", "backtesting");
            string algorithm = "EMATest";       // put this algorithm is QuantConnect.Algorithm.CSharp folder
            Config.Set("algorithm-type-name", algorithm);

I hope this helps.

Bizcad aka Nick Stein

Contributor

bizcad commented Nov 3, 2015

@gururise If you copied the code from @chrisdk2015 Optimization project on github, my guess is your problem is on line 154 of Program.cs

154 _resultshandler = new DesktopResultHandler ();

Try changing all the DesktopResultHandler to ConsoleResultHandler. DesktopResultHandler expects a Desktop application to write results to. In the console application there is no desktop.

Also take a look at my take on a complete solution including Lean and all it's sub-projects. This link is also in my reply above. I changed DesktopResultHandler in three places to an OptimizationResultHandler with some commented lines next for ConsoleResultHandler. If you want to see the results flash by in the console comment out the OptimizationResultsHandler and uncomment the ConsoleResultsHandler.

The code for the OptimizationResultsHandler is in the Engine\Results folder and is based upon ConsoleResultsHandler.

I run in Windows/VisualStudio, so I place a breakpoint about line 75 of Program.cs at the top of the LaunchLean function and step from there to find my error.

            Config.Set("environment", "backtesting");
            string algorithm = "EMATest";       // put this algorithm is QuantConnect.Algorithm.CSharp folder
            Config.Set("algorithm-type-name", algorithm);

I hope this helps.

Bizcad aka Nick Stein

@jameschch

This comment has been minimized.

Show comment
Hide comment
@jameschch

jameschch Jan 23, 2017

Contributor

This is a fairly old ticket and it appears work has stalled. I have recently overhauled the LeanOptimization project here:

https://github.com/jameschch/LeanOptimization

Anyone still interested in this functionality please feel free to provide feedback, open issues etc.

Contributor

jameschch commented Jan 23, 2017

This is a fairly old ticket and it appears work has stalled. I have recently overhauled the LeanOptimization project here:

https://github.com/jameschch/LeanOptimization

Anyone still interested in this functionality please feel free to provide feedback, open issues etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment