HWs and Project

Final Project

Here's the write-up of the final project. I'll go over it in class and answers any questions you have there or by e-mail. Project due Friday, May 5th 2017.

HW 3

Homework 3 (for the week of April 3rd - due date is Sunday April 9th):

So here's something to keep you busy while I tour the east coast colleges with my daughter. Abi will be around during normal class hours to help you out.

We are going to be playing with CASTRO and the Kelvin-Helmholtz problem they have set up. This is taken from: https://arxiv.org/abs/1111.1764

Log into cori.

Start by copying the directory /project/projectdirs/astro250/GitCode/Castro/Exec/hydro_tests/KH to your $HOME directory. This will allow you to see it from yt which we will need for later.

cp -r /project/projectdirs/astro250/GitCode/Castro/Exec/hydro_tests/KH $HOME/

cd into the directory

cd KH

And now you are going to have to switch over to the gnu compilers. This you can do with a:

module swap PrgEnv-intel PrgEnv-gnu

and now you should be good to go with the version of the code here — Castro2d.gnu.MPI.ex

(you can also type make if you have any problem with the below to make sure you are set up right)

I want you to run the problem once with no refinement, and once with at least two levels tagged for refinement in any manner you like. As Abi mentioned in class you will need to have MPI and the gnu compilers set up, email her if you have trouble with this.

You can adjust the problem parameters using two files:

In inputs.2d: Here you will need to adjust the maximum number of refinement levels allowed by changing:

amr.max_level  (when this = 0 points to no refinement and = # allows for #+1 levels).

You will also need to play with the parameters in the probin file. Here you can adjust the fluid parameters, such as the density of each layer as well as control how you tag for refinement. I want you to fill in the tagging criterion however you see fit.

&tagging
     dengrad = (tagging criterion for when density changes, at what value of the gradient of density should we tag for refinement)
     dengrad_max_level = (how many levels of refinement can we reach through the density gradient criterion, this must be <= your amr.max_level)
/

You can also play with the equivalent velgrad, and velgrad_max_level.

This simulation should run on 1 node and 32 cores for < 10 min. Please don't refine past 3 levels just for computer resources sake. You can grab an interactive node to do this, it should only take a few minutes to run.

salloc -N 1 -p debug -t 00:30:00 -C haswell

Run the problem while pointing to you inputs file:

srun -n 32 ./Castro2d.gnu.MPI.ex inputs.2d

until t=2.0s. This is controlled with the stop_time variable in the inputs.2d

Plot up the final plt_file using the yt notebook (see the instructions above). I would like you to upload two images into your GitHub repository, one with the grid displayed and one without for both your unrefined case and any refinements you try. Feel free to have fun with yt, play with some of the new color maps for matplotlib, etc. I would also like a text file describing your choice of refinement criteria. I've created a simple yt notebook you can fire up to look at two of the runs Abi has done already in /project/projectdirs/astro250/python/CASTRO.ipynb

Note: in the problem above the instability is perturbed with a single frequency, so this case should be linear and remain smooth. However, the AMR seems to be seeding small scale perturbations that you may see as you refine the problem. IF you would like to examine a non-linear Kelvin-Helmholtz (which will look a lot more interesting) take a look at problem 5, taken from a group here at Berkeley: https://arxiv.org/abs/1509.03630 This problem can be run the same way but using the inputs.2d.p5 and corresponding probin.p5 parameters.

---

Something that came up in class today. You'll either have to move your plt files to a new directory before running again with refinement, or edit your input file so they plot under a new name instead of overwriting your previous run. You can do that in inputs.2d, by changing the root names of the chekpoint and plotfiles:

# CHECKPOINT FILES

amr.check_file      = chk_   # root name of checkpoint file

# PLOTFILES

amr.plot_file       = plt_       # root name of plotfile

HW 2

Please submit the following homework assignment by creating a directory in the homework2 repo titled 'yourlastname_hw2'. Be careful with your add and commits, make sure you are just pushing changes made to your own directory!

Homework 2:

Take the code jacobi.f90 (from /project/projectdirs/astro250/class/code/hybrid cp this over to your home directory and compile it with the command "make") or write one of your own which is a hybrid MPI and OpenMP code. A very simple example would be something which does a numerical integration. For example you could modify the following MPI code to have an OpenMP part in the function integral:

https://www.dartmouth.edu/~rc/classes/intro_mpi/Numerical_integration_example.html

Here you need to remember to set up a pragma on the loop and do the reduce instruction (perhaps with some private variable) to make that work. Google around and find something to play with...

You can even have it do something completely silly like add, exponentiate and multiply a bunch of numbers.

Whatever you do the code should run for approximately 5 minutes when running on 2 cores - you'll probably have to modify jacobi.f90 to do more steps. I do not care that you are doing something useful or sensible - if you are doing an integral just make your dx really small so you can see the cost benefit from below.

*However, I do care that you get the same result (to within machine precision) regardless of how it is run below and this needs to be printed out.

For now, lets compile on the cori login nodes and just submit to the haswel partition of the machine. KNL's will come next week (and how to optimize for either/both).

Create a submit script so that the standard out gets saved to either the job file or a file you name. Then, to start, I would like you to run it straight up in MPI on 2 nodes using all the cores you have available. You should have your srun command prefixed with time so we can see the output.

Then I would like you to try 6 more example runs, all in the same submit script, with different MPI tasks and OpenMP threads so that we can see the difference. Thus you should change OMP_NUM_THREADS and the srun commands each time. Do whatever you want but I want the extremes covered which is the above, pure MPI on everything, to just 2 MPI tasks with the rest being fully OpenMP on each node. You can certainly play with the hyperthreads as well.

You'll need to push your code, any compile options and the submit script to the git repo.

The homework will be due Friday March 3.

HW 1

Here's the link to the first homework assignment.

https://classroom.github.com/assignment-invitations/73ff690eb33b72c056ae80e4316f42b6

This will work on edison only for now!

Please try to fire this up. I had to change the path for everything to point to my scratch directory on edison. Give it a whirl and holler at Abi or I if there are any issues.