io-watchdog - The IO Watchdog. The IO Watchdog is a facility for monitoring user applications, most notably parallel jobs, for "hangs" which typically have a side-effect of ceasing all write activity (IO) in a cyclic application (i.e. an application that writes something to a log or data file during each cycle of computation). The io-watchdog attempts to monitor all write activity coming from an application and triggers a set of user-defined actions when IO has ceased for a configurable timeout period. The IO watchdog consists of a LD_PRELOAD library (the interposer) which intercepts calls to various output-related calls in libc, along with a watchdog server which wakes up periodically and ensures that the application has written something during the last timeout period. If not, the watchdog server issues a warning on the application's stderr, and invokes all user defined actions, which could include running a debugger on the application, sending email to the user, etc. Set up of the LD_PRELOAD library is facilitated with either the io-watchdog(1) utility, or a SPANK plug-in for SLURM which adds a new --io-watchdog command line option to srun(1). To enable the io-watchdog SLURM plugin, the following line must exist in /etc/slurm/plugstack.conf: required io-watchdog.so The io-watchdog supports the following tunable parameters: timeout The watchdog timeout. Default = 1 hour. rank The MPI rank for which the watchdog runs if a SLURM job. actions A list of actions to run on watchdog trigger. target A pattern match for target of io-watchdog if running multiple applications in a pipeline or single job. These may be set on the command line, or in an io-watchdog configuration file. Configuration files that are read automatically if they exist are /etc/io-watchdog/io-watchdog.conf System defaults ~/.io-watchdogrc User defaults. A config file may also be specified on the command line to override the default location of the user configuration. See "io-watchdog --help" and "srun --io-watchdog=help" for more information.