Wall clock management

[joaander]

Original report by me.

---

HOOMD's current limit_hours is a poor way to manage wall clock times. We need a simpler more comprehensive solution. One that allows job scripts to set environment variables to control the whole job run time from the start of the script execution.

[joaander]

Add HOOMD_WALLTIME_STOP env var

Set this env var to a unix epoch time value to force hoomd runs to stop
at that time. Use it with limit_multiple to force stops at specific time
steps. run() uses predictive timing based on the current average TPS to
make sure the run will stop before the given time.

To allow for multi-stage runs, when this env var is set, time limited
runs will throw an exception. The user script needs to wrap the runs in
a try, except block to ensure proper cleanup at the end of the scriipt.

There is no documentation for this yet, that will come later. Also, it
currently throws StopIteration. This is a problem if the run is
happening in a for loop. We need a custom exception type to avoid this
problem.

refs #47

[joaander]

TODO: Need to create a custom exception.

[joaander]

Also, for later reference when writing the docs. Here is the syntax for setting the env var in bash:

HOOMD_WALLTIME_STOP=$((`date +%s` + 12 * 3600))

[joaander]

Add WalltimeLimitReached exception

This exception is thrown when HOOMD_WALLTIME_STOP is used to limit the
length of hoomd runs. User scripts should catch this exception in order
to provide necessary cleanup of jobs. Using a custom exception avoids
potential pitfalls or confusion over built in exceptions.

refs #47

[joaander]

Document HOOMD_WALLTIME_START

Add the first level of documentation for HOOMD_WALLTIME_START. It is
documented in the run() command itself.

refs #47

[joaander]

Add HOOMD_WALLTIME_STOP env var

Set this env var to a unix epoch time value to force hoomd runs to stop
at that time. Use it with limit_multiple to force stops at specific time
steps. run() uses predictive timing based on the current average TPS to
make sure the run will stop before the given time.

To allow for multi-stage runs, when this env var is set, time limited
runs will throw an exception. The user script needs to wrap the runs in
a try, except block to ensure proper cleanup at the end of the scriipt.

There is no documentation for this yet, that will come later. Also, it
currently throws StopIteration. This is a problem if the run is
happening in a for loop. We need a custom exception type to avoid this
problem.

refs #47

[joaander]

Add WalltimeLimitReached exception

This exception is thrown when HOOMD_WALLTIME_STOP is used to limit the
length of hoomd runs. User scripts should catch this exception in order
to provide necessary cleanup of jobs. Using a custom exception avoids
potential pitfalls or confusion over built in exceptions.

refs #47

[joaander]

Document HOOMD_WALLTIME_START

Add the first level of documentation for HOOMD_WALLTIME_START. It is
documented in the run() command itself.

refs #47

[joaander]

Merged in restartability (pull request #45)

Enable restartable jobs more easily

fixes #45, #46, #47