Running KHARMA

To run a particular problem with KHARMA, you can usually simply invoke kharma.host for CPU compiles, and kharma.cuda for Nvidia GPUs. The problem initialization and parameters are specified in a .par file:

$ ./kharma.host -i pars/orszag_tang.par

KHARMA benefits from certain runtime environment variables controlling CPU pinning and occupancy, which can be handled by the wrapper run.sh. The script is not intended to be terribly robust, but rather to serve as a guide to running KHARMA on different machines in an efficient way.

Note that some MPI libraries may require that kharma.{host,cuda} be run with mpirun -n 1 ./kharma.host even when invoking a single process. NVHPC in particular simply hangs if invoked without an external MPI environment.

All the parameters specifying a simulation are provided by a single input "deck" in the .par file. Sample input files corresponding to standard tests and astrophysical systems are included in pars/. The same KHARMA binary can run any .par file, regardless of spacetime geometry, problem initialization, etc.

KHARMA will attempt to guess many parameters if they are not specified, e.g. boundary conditions and coordinate sizes for simulations in spherical polar coordinates, or interior boundary locations for black hole simulations based on keeping 5 zones inside the event horizon. Most of the core inferences are done in the function FixParameters in kharma.cpp, and most of the default values are specified in the Initialize functions of their respective packages, e.g. in grmhd/grmhd.cpp.

Any parameter can be overridden from the command line, useful for scripting scaling tests, regression tests, etc. Simply pass the full parameter path and value, e.g., parthenon/output0/dt=1 to change the parameter dt in block <parthenon/output0> to 1.0. These overrides must be passed after the initial parameter file argument. For example, to turn on verbose console output:

$ ./kharma.host -i pars/orszag_tang.par debug/verbose=1

Parameters should not be preceded with the usual dashes or other flag characters.

Parthenon will warn (but not crash) if a new parameter is specified from the command line which is not present in the parameter file. Parameters which are unused or unknown to Parthenon or KHARMA will be silently ignored.

Outputs

Nearly all problems will write out the state of the simulation every once in a while, for plotting or analysis after a simulation is complete. The cadence and contents of these "dump" files are specified in the parameter file; all outputs are usually listed together at the end. The fluid state is generally output to HDF5 files named problem_name.out0.NNNNN.phdf, with the sequence number NNNNN counting up incrementally from 00000. Additionally, some problems compute reductions (total energy, accretion rate, etc.) and output these to a text file, problem_name.hst.

Output files are split first by mesh block ID, then vector index if present, then by cell in k, j, i order. Since data is split by block, reading the file for e.g. plotting, analysis, etc. means either treating every meshblock separately, or mapping their contents onto a single global mesh. Since these operations are tedious to reimplement, I recommend copying or using existing solutions:

• Parthenon provides a small python package, parthenon_tools, designed to read these output files and produce a few basic plots.
• For reading KHARMA output specifically, the pyharm package provides reading, calculation, and plotting tools.

Restarting

In addition to the analysis outputs, most long-running problems will output "restart" or "checkpoint" files. These are named similarly to the science outputs, with the pattern problem_name.out1.NNNNN.rhdf (note the changed file extension as well as the output number).

A simulation can be resumed very simply from such a file. E.g. to restart a torus simulation from the 200th restart (by default corresponding to 20,000M of simulation time):

$ ./kharma.cuda -r torus.out1.00200.rhdf

Note that -r is incompatible with -i. All of the original problem parameters are saved to .rhdf files, and Parthenon will read them automatically when restarting a simulation, so no parameter file is necessary.

Parameters can be overridden when restarting, using exactly the same syntax as when running a new problem. Needless to say, be very careful with this -- many potential changes (e.g., mesh size) will at best produce gibberish, at worst segmentation faults.