Darwin - %age and ETA's during backtesting, as well as mid-generation resuming

[glennfu]

Problems I hoped to solve:

1. A backtest with a very high population might take more than 24 hrs to execute. If you're 80% done, but need your computer back for other things, you might really struggle with the decision on whether or not to let the generation finish, or kill it.
2. A sim might be broken (see crossover_vwap hangs indefinitely on unstable, succeeds on master #1249) or maybe taking longer than others, or maybe it's just taking a while and you'd like to know more. Current output gives no information about any of this.

To solve resuming mid-generation:

I now generate a token for the "run" that looks like "backtest_201801300356" where the number there is the timestamp. Inside it are nested files and folders related to the data. The pro being no more scattered millions of files in a single simulations folder. Now everything for the entire "run" is stored in one place.

To resume, you just add --population_data=backtest_201801300356 to your previous command and it'll re-start on the correct generation and sim set. So if you were at 87/100 sims and you closed it and restarted, you'll still be at 87/100 sims remaining.

As another benefit to this is that as soon as a sim is done, you can view its info and results within its own file within its generation folder, including the html output.

The only downside here is that this is not backward compatible with the old population_data param so any old runs you may have stored will no longer be resumable. I could revisit this if people think it's a real problem but I didn't want to clutter the codebase with multiple solutions to the same problem.

To add meaningful console output:

Instead of printing out the command along with the sim number, the command info is added to its sim file in the generation folder immediately, and now the output is made to be something more useful.

1. Start of the run, and at the end of each generation, timestamp and duration is printed.
2. During a run, you get info about how many sims are running, and the completion of each active run, and an ETA for the slowest currently running.

Done: 18, Active: 8, Remaining: 4, Completion: 66.4%, Slowest sim ETA: 1m 22s  18: 44.0%, 20: 26.7%, 21: 28.0%, 22: 26.4%, 23: 23.9%, 24: 13.7%, 25: 28.7%^C

3. Once a generation gets near completion, you'll get an ETA for every remaining sim. This can be useful for figuring out what's up with some particularly long running sims.
4. Like during trade/sim where it keeps updating the same line over and over, this does the same thing to not be overly noisy.

Final thoughts:

I don't actually think the PR should be simply accepted as is. I'm hoping for some people to really hammer at this and give some feedback and ideas on how to give it some better polish. I was originally hoping that I could give an ETA for the entire run early on, but considering some sims may take much longer than others, I can't figure out a way that doesn't involve lots of guesses with averages to compute and that'll just annoy people because it'll be wrong.

At any rate, I feel like there's more opportunity for useful output there but now that I've hit a solid stopping point I'm low on ideas. Let me know what you think!

[dauv]

Any reason for keeping this outcommented code (and the other blocks) in the PR?

[glennfu]

This was part of an earlier attempt to measure time by first figuring out how many periods are being processed, and then measuring time per period. I quickly found out that time per period can vary based on lots of different things. However since that part worked, I was hesitant to get rid of it entirely in case someone else here had an idea of how to leverage it.

[glennfu]

Couple of thoughts:

1. It should give some clear indication of how to resume a run. Maybe by listening to kill signal and giving a copy-and-pastable command to do so.
2. Next to "Slowest sim ETA" it could also show "Current best balance"

[glennfu]

Both these last two thoughts are now implemented.

Here you can see in generation 1 it was spitting out the best balance mid-generation of runs that had completed so far. In generation 2 you can see it's spitting out a progress report of actively running sims.

At the bottom after I hit Ctrl+C you can see it gives you a copy-and-pasteable command you can run to resume the previous backtest.

[glennfu]

@dauv @DeviaVir please give this another look now. I think it's ready!