Run the simulations

[rosemckeon]

May be worth looking at how to schedule these on the server by making a repeating job that sets itself going again once complete.

https://stackoverflow.com/questions/18306362/run-r-script-from-command-line
https://linux.die.net/man/1/rscript
https://www.interserver.net/tips/kb/task-scheduling-linux-command-line/
https://www.taniarascia.com/how-to-create-and-use-bash-scripts/

https://askubuntu.com/questions/8653/how-to-keep-processes-running-after-ending-ssh-session

[bradduthie]

@rozeykex Should work -- can also use a BASH loop, I think.

[rosemckeon]

Noting here what we discussed today about the simulations we'll run @bradduthie (once I've found the sweet spot for the NULL simulations I'll let you know here too).

• Run NULL tests first to see what pattern emerges if there is no advantage to being polyploid and all individuals can mate freely with an equal chance of success.
• Vary ploidy_growth_benefit.
• Vary disturbance_freq.
• Vary disturbance_mortality_prob
• Vary pollen_range
• Vary clonal_growth

Consider varying seedling_selection_constant later to change the strength of selection against size.

[rosemckeon]

Hi @bradduthie, I'm starting to get some data in now. Haven't got as many runs in over the weekend as I wanted, but I think I've got a good stable NULL point now that doesn't take too long to run. I've saved it here as a script you can run with Rscript simulations.R from a unix/BASH prompt. There's an analysis file with some quick plots to check the pop levels in the data once it's finished.

Seed survival seems to be the key parameter I've had to tweak gently to get population levels right. In my tests, over the weekend, I had this a little higher at 0.535 and all 4 sims got to 100 nicely but it looked like some were about to boom and go wild. The set of data I pushed yesterday was with that set a little lower (0.5325) and there was too much extinction. I have some running right now at 0.534 and they seem to be doing well I've got 1 of 2 sims that have reached 200.

[bradduthie]

Hi @rozeykex -- sounds good. I'll start processes of this in parallel and see how it goes!

[rosemckeon]

Oh, @bradduthie! I should also have said: I have runs <- 2:4 being covered. If you do 5+ and let me know your top limit then we can make sure we don't push conflicting data and log files. I've kept sims set as 4 as it's a nice number to have in one data file for facets and filesize.

[bradduthie]

Ah! @rozeykex I was actually just copying the whole directory locally four times and running in parallel to get the data. Should I also try a 5:7 assigned to runs to see how that works?

[rosemckeon]

Yep! @bradduthie I've made it so it outputs everything pretty neatly. Rather than copying the folder many times, to run many in parallel you could just copy the simulations file and have different values assigned to runs. Then all the output will just organise itself within the analysis/data/ folder.

[bradduthie]

@rozeykex Okay, cool! Do I just need to change runs <- 5:7, simulate, and then push then?

The local version is running fine -- Simulation 3 Generation 108.

[rosemckeon]

@bradduthie sorry, been away from the computer! Yep. That will make sure the .rds files and log .txts that you push are all named differently to mine. The data objects reference the logs that go with them.

[bradduthie]

@rozeykex Okay, will start on this tomorrow morning!

[rosemckeon]

@bradduthie I've added runs 8:11 and set 12:15 going with a bit higher seed survival.

[rosemckeon]

Notes from today's meeting with Mario:
NULL sims need to model a perennial iteroparous plant system more accurately and remove the seed bank so genomes created in the past are not appearing later to confuse patterns. We'll do this by:

• Having seeds germinate or die (no dormancy).
• Having adults survive the winter.
• Having adults survive at higher rates than juveniles.

From logs, current seedling_selection_constant of 0.25 equates to roughly around 27% seedlings surviving.

[rosemckeon]

Hi @bradduthie, welcome back. I'm starting to get some data on the sims that test everything individually. I've been away for the weekend so getting my head properly in the literature for the realistic system now too. Please feel free to run any of the scripts in the costs/benefits folder in /simulations. If you set runs > 5 then the files we push shouldn't overlap.

I think costs/triploid_mum_prob/sterility-200.R has the most tests in and I might be running a risk of not completing all of them. It might be even better if you run tests in reverse order? - let me know if you want me to set up some scripts like that.

Also, FYI I've bitten a bullet and updated my github username to something perhaps more professional!

[bradduthie]

Hi @rosemckeon -- good to be back! Okay, I've pulled the branch 'rose' and will start running some simulations. I'll set runs == 10 to start with so there is no overlap. I'll happily run whatever you want in parallel on my office and home machines, if you can just send a quick script to source? I should have at least 5 cores open on my office computer, and one on my home computer.

[rosemckeon]

@bradduthie great! I'll push some scripts for you now, with the sims all running back to front, and tag you in the commit. I'll set up the run numbering too. I've found when the processes get killed early it's usually due to lack of memory rather than processing power. This whole thing has got me looking at 32GB RAM upgrades....

[bradduthie]

@rosemckeon Whoa! That is quite memory heavy. Too late for this now I suspect, but if you build on these simulations for a future project, it might make sense to get rid of unused memory during the simulation (e.g., by printing prior generations to a file, then removing them from the R environment). The gc() function might be able to free up a bit by garbage collection.

I've got 16 GB to work with here. I'll start with one run to see how things look, then start trying everything in parallel.

[rosemckeon]

@bradduthie I was wondering if there was something I could do differently like splitting the files up into smaller chunks! I don't really want to risk breaking things by attempting last minute speed improvements but it's tempting... I'll definitely take a look at gc() if I get to expand on this for another project.

[rosemckeon]

@bradduthie I've got 16GB on the server and it manages 4 in parallel most of the time, with occasional process deaths.

[bradduthie]

@rosemckeon Yeah, splitting files into chunks is often a good solution. I typically try to avoid storing information (e.g., old individuals) that is no longer needed in the working memory. I agree that it's probably not worth the work and risk at this point, so no worries. We should be good with some simulations on both your server and my desktop(s)!

The gc() is actually easy to work with. It doesn't require any additional arguments, and can be expressed wherever you want to free up some unused memory (e.g., at the end of each generation. It's unlikely to free up a ton of memory, but it might make things just a bit more manageable for not a whole lot of time loss.

[rosemckeon]

@bradduthie So I can just do gc() whenever I clear the current generation object? the reset parameter doesn't need to be TRUE?

[bradduthie]

@rosemckeon Yeah, I think that this should work. I think that R does this automatically to some extent, but it might help a bit.

[rosemckeon]

@bradduthie done :) Can you let me know if it helps enough to interrupt my simulations and start them again? I don't really have the free memory at the moment to test it.

[bradduthie]

@rosemckeon Will do! Just pulled and am now running 'inbreeding-200-brad.R'. Will give it a few minutes, then run the other two scripts.

[bradduthie]

@rosemckeon Now running growth-200-brad.R and selfing-200-brad.R. The simulation inbreeding-200-brad.R is currently on generation 66.

[rosemckeon]

These are happily ticking along now and managed in another repo.