Pop ensemble hof

[bartleyn]

What does this PR do?

Sets up internal testing for population ensembles.

Where should the reviewer start?

Looking at main() for the added arguments necessary to test.

How should this PR be tested?

Calling tpot.py with a set of parameters, including new parameters -n (random sample size), --iters (number of trials to run), --hof (whether or not to use the hof versus the pop in ensemble testing). Instead of using -o for exporting the code, -o now specifies where to print the logbook (with the stats).

Any background context you want to provide?

I rolled in some of the testing changes I made in a separate testing script into the main() for easier batch testing.

What are the relevant issues?

[you can link directly to issues by entering # then the number of the issue, for example, #3 links to issue 3]

Logbook will be tab delimited and should be manually cleaned up.

Screenshots (if appropriate)

Questions:

• Do the docs need to be updated?
  No
• Does this PR add new (Python) dependencies?
  No

[rhiever]

Wow, this required a lot more code than I expected! Was the custom implementation of eaSimple necessary? And all the consensus_two etc operators?

[bartleyn]

You're right. I neglected to trim out some code. I'll clean it up, run a test, and submit a commit shortly.

[bartleyn]

The only thing that changes here is that you shouldn't pass a verbosity parameter. The logbook with all the generation data will write to file after the eaSimple finishes, and using verbosity seems to flush the logbook output to stdout rather than to file.

[rhiever]

Cool beans. Can you also please make sure that it's using this accuracy
metric: https://github.com/rhiever/tpot/blob/master/tpot/tpot.py#L1192

On Sun, Mar 27, 2016 at 12:55 AM, Nathan notifications@github.com wrote:

The only thing that changes here is that you shouldn't pass a verbosity
parameter.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#121 (comment)

Randal S. Olson, Ph.D.
Postdoctoral Researcher, Institute for Biomedical Informatics
University of Pennsylvania

E-mail: rso@randalolson.com | Twitter: @randal_olson
https://twitter.com/randal_olson
http://www.randalolson.com

[rhiever]

Going to close this now since it didn't seem to work out, but let's keep this idea in mind for future TPOT iterations.