Freeprocessing in situ visualization
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
contrib add enzo/yt example python script. Apr 29, 2014
doc fix some typos/edits/etc. (thanks Jens!) May 2, 2014
fortran-test move fortran test programs out of root dir. Mar 5, 2014
m4 gbs: provide mechanism to build python-based sources. Mar 10, 2014
processors fix python processor undefined symbols. Jun 27, 2014
pvsim add example as to how to do PV in situ. Mar 5, 2014
.gitignore add 'nositu' to build. Apr 29, 2014
.travis.yml travis: enable/test python. Mar 19, 2014
COPYING first pass at an autotools build, still broken in many ways. Mar 5, 2014
LICENSE the symbiont is covered by the LGPL. Mar 5, 2014 add 'nositu' to build. Apr 29, 2014 document the craziness that is BlueGene. Mar 12, 2014
acinclude.m4 gbs: provide mechanism to build python-based sources. Mar 10, 2014 add new freeprocessor which implements the original functionality. Mar 5, 2014
compiler.h Free memleaks, particular on intel. Mar 5, 2014 gbs: more portable way to figure out the LIBdir var. Mar 12, 2014
csv.c write a 'correct' CSV, i.e. the new way from Burlen. Mar 5, 2014
debug.c fix clang warning Mar 10, 2014
debug.h shift unsigned numbers. Mar 5, 2014
echo.mpi.c reimplement echo to be mpi-based. Mar 5, 2014
enzo.mpi.c fix typos in comments Mar 5, 2014
evfork.c C program to set LD_PRELOAD then fork/exec. Mar 5, 2014
evfork.h C program to set LD_PRELOAD then fork/exec. Mar 5, 2014
forkenv.c situ: set the right environment variable. Mar 5, 2014
fp-png.h add a library for nek which writes out PNGs. Mar 5, 2014
fproc.c grid_size -> metadata Mar 5, 2014
fproc.h grid_size -> metadata Mar 5, 2014
h5.c grid_size -> metadata Mar 5, 2014
modified.c new code for checking file modification Mar 5, 2014
modified.h new code for checking file modification Mar 5, 2014
nek5k.mpi.c add a library for nek which writes out PNGs. Mar 5, 2014
open.mpi.c instrument MPI file open, + assert functions exist Mar 5, 2014
override.c fortran example/test code. Mar 5, 2014
parallel.mpi.h add missing stdlib header to parallel.mpi.h. Mar 5, 2014
parallel.mpic fix python processor undefined symbols. Jun 27, 2014
parenv.mpi.c mpi programs for debugging env issues Mar 5, 2014
png.c png: fix bug in error case Mar 5, 2014
posix.c ignore readonly again... Mar 5, 2014
runner.mpi.c mpi programs for debugging env issues Mar 5, 2014
setenv.mpi.c mpi programs for debugging env issues Mar 5, 2014
simplesitu.c fopen: ignore read-only opens. Mar 5, 2014
testmodified.c missing file from last commit! oops. Mar 5, 2014
vis.c provide the ability to log to a file. Mar 5, 2014
vis.h provide the ability to log to a file. Mar 5, 2014
writebin.mpi.c new code for checking file modification Mar 5, 2014

This is a data transfer symbiont. It is essentially 'tee' in library form.

To use it, LD_PRELOAD the library ( into your program. At that point, it looks for all of your write calls and does its magic.


The library looks for a library named situ.cfg. This library defines what libraries should be loaded. The general format is basically like awk, except that instead of specifying the code to run, you specify a library that implements that code:

pattern { exec: /path/to/ }

Where pattern is a shell glob and is a library implementing a pre-prescribed interface (see LIBRARIES). An example:

  *header* { exec: ./ }
  *.data { exec: ./ }
  *tosend* { exec: ./ }

This defines three processing elements, libmine will be used whenever a filename which matches *header* or *.data is accessed. Both instances get called. The order they are called is undefined.

Sharing internal memory

If a single library is given in multiple patterns, the matches share private variable instances. In the above example, both instances will access a single set of shared variables. This allows you to (for instance) parse the program's header output and communicate such information to the part of the code which instruments the binary outputs.

If you actually want two instances of a library, with unique internal variables each time, just rename one of them. The equivalency is based purely on the name of the libraries, so libraries of different names---even if they are the exact sample implementation---will not share private data.


Libraries need only export one function:

  1. void exec(const char* fn, const void* buf, size_t n)

It is called anytime a file is written to. The filename given in the first argument is the exact filename (not, for example, the pattern given in the configuration file).

The symbiont attempts to take care of the issue of partial writes; there should be a 1-1 mapping between program writes and exec invocations.

Libraries can optionally implement two additional functions:

  1. void metadata(const char* fn, const size_t dims[3], int), and
  2. void finish(const char* fn)

metadata is called directly before exec, in the case that the simulation knows metadata for the impending write. This is only possible with some self-describing transport mechanisms. If your code does not utilize IO middleware, this function will never be called. In this case, you have to provide any needed metadata yourself. We recommend parsing a config file the first time exec is called.

finish notifies you when the program is finished with a file or pattern. Good time to clean up resources.


  • LIBSITU_DEBUG: used to selectively enable debug information. See below for more info.


There can be issues forking processes. This includes using system. Therefore, the library unsets LD_PRELOAD during initialization, so that it will not instrument any child processes.

For the most part, this is what you want. However, this may cause problems if you have a complicated loading procedure ("mpirun" and "valgrind" come to mind). To get the code to run under MPI, you want the mpirun program to set the environment variable for you, instead of setting it before mpirun is invoked. OpenMPI's mpiruns let you do this with -x, e.g.:

mpirun -x LD_PRELOAD=./ -np 4 ./myprogram

For MPICH, this option is -env and the setting should lack the = character:

mpiexec -env LD_PRELOAD ./ -np 4 ./myprogram

BlueGene's are even more unique:

runjob --envs LD_PRELOAD=./ --exe ./myprogram

Debug Channel Settings

The symbiont has a set of unique debug channels. Generally these are all related to a specific feature, such as investigating when files are opened, or when data is transferred, or other information.

In addition, each channel has a set of classes, which might loosely be thought of as severity. So you can, for example, turn on only the warnings from a particular channel, or only the errors.

The way one configures these channels is through the LIBSITU_DEBUG environment variable. The general form is channel name=[+-]class[,...], e.g.:


which says to enable warnings on the channel opens. Multiple class specifications are separated by commas:


Multiple channels are delimited by semicolons:


The currently defined channels are:

  • opens: all open-esque calls, along with their close counterparts.
  • freeproc: loading and using libraries
  • writes: all write-esque calls. (CAUTION: spammy)
  • posix: trace POSIX I/O calls.
  • hdf5: HDF5 calls of interest.

The currently defined classes are:

  • err: errors which almost certainly indicate a bug in either the symbiont or the host program.
  • warn: generally anomalous conditions which might not be a problem, such as an internal table filling up.
  • trace: notification when a call occurs, in a manner similar to strace(1) or ltrace(1).


It seems that the executable needs to be built as position-independent code for this to work.

if the code is not position-independent, then __attribute__((constructor)) functions do not get run and thus our function pointers never get initialized.

Fortran may work a bit differently, doesn't seem required there.