A Go project for generating defined HPC workload characteristics (CPU usage, RAM usage, etc.) on a system to facilitate granular monitoring.
This project is written primarily in Go and requires Go v1.15 or later to compile. Cgo is also used, so you'll need a recent version of gcc.
To build, you can just type go build, and Go will handle everything.
Alternatively, if you have Make installed, you can just type make. Both of
these methods produce an executable binary called hpc-workload-generator.
To view help, run ./hpc-workload-generator -h.
$ ./hpc-workload-generator -h
Usage of ./hpc-workload-generator:
-cgroups-root string
path to the root of the cgroupsv1 hierarchy (default "/sys/fs/cgroup")
-d uint
debug level (0-6) where higher numbers have higher verbosity (default 4)
-i string
path to the schedule file to execute, or empty for stdin
The -i option is the most critical and specifies the path to the input file
from which to read the schedule of events. If not specified, the program will
read from standard input.
The -cgroups-root option allows you to change the default location of the
cgroupsv1 hierarchy if it happens to be mounted somewhere unusual. This can also
be handy when using a container in case you want to mount the hierarchy in a
different location for the sake of the container.
The -d option allows you to set the debug level. The default (4) is fine for
most cases, but you can bump it up to 5 or 6 if you want more verbose logging.
A Docker container is provided for systems where that is more convenient (such
as a Kubernetes cluster). You can build it manually using the provided
Dockerfile, or just pull the pre-built copy from Docker Hub. Example usage
follows:
docker run -t --rm \
--mount type=bind,src=/sys/fs/cgroup,dst=/sys/fs/cgroup,readonly \
--mount type=bind,src=`pwd`,dst=/data \
phphavok/hpc-workload-generator -i /data/example.schedule
We specify -t so that we're allocated a pseudo-terminal which makes the
logging output look nice and formatted. The --rm option automatically cleans
up the container on exit. The first mount command passes through the cgroupv1
hierarchy (/sys/fs/cgroup) on the parent system to the same location within the
container. By default, Docker will often have some of the cgroup hierarchy
present within the container, but not all of it. This application will need to
see the full hierarchy, so this read-only bind mount takes care of that. If you
run into issues with mounting over the existing hierarchy within the container,
you can change the target to some other location and then pass the
-cgroup-root option to the program to accommodate that change. The second
mount command just passes through the current working directory (assumed to be
this repository's root) so that schedule files can be accessed and ran on from
within the container. The entrypoint to the container is the program itself, so
you can just pass its parameters to the run command.
You can use Singularity (e.g., on an HPC cluster) to run the above Docker container from Docker Hub.
singularity run \
-B /sys/fs/cgroup:/sys/fs/cgroup \
-B `pwd`:/data \
docker://phphavok/hpc-workload-generator -i /data/example.schedule
The schedule of events is an input file that the hpc-workload-generator program reads. The format of the file is one event per line, where an event line is a comma-separated list of values. The first value indicates at how many seconds into the runtime the event should fire. The second value is the name of the event module to run. All remaining values are arguments to the specified event module. An example schedule of events follows with commentary:
4, cpuload, 1, 70, 12
8, cpuload, 3, 70, 12
12, cpuload, 5, 70, 12
16, cpuload, 7, 70, 12
This schedule will execute the cpuload module four different times at 4 seconds, 8 seconds, 12 seconds, and 16 seconds, respectively. Each execution has different potential settings. Note that modules can and will be executed in parallel if you specify multiple modules at the same second offset (or if one module is still running when a later module begins).
This particular schedule will cause CPU number 1 (0-indexed) to be taxed at roughly 70% for 12 seconds starting at 4 seconds into the runtime of the program. At 8 seconds into the runtime, CPU number 3 will start being taxed at 70% for 12 seconds. Note that at this point, only 4 seconds have passed since CPU 1 started being taxed, so it still has 8 seconds left of being taxed. Thus, CPU 1 and 3 will be taxed in parallel at this point. The rest of the schedule follows a similar scheme.
Read more about the supported modules and their parameters below.
The cpuload module generates simulated load on a CPU. The module has three
arguments.
The first argument is the index of the CPU to generate load on. It is 0-indexed
while programs like htop display the CPUs as 1-indexed, so you'll have to do
some mental gymnastics when looking at live results. Furthermore, the CPU index
is automatically mapped into the program's active cgroup. Suppose you're running
the program on a node that has 16 CPUs, 8 of which have been reserved for your
node via a cgroup. It's possible that you'll be reserved CPUs that aren't
contiguous. That is, instead of getting CPUs 0 through 7, you may get CPUs 0-3
and then 8 through 11. You won't know this ahead of time when writing a schedule
of events though. Therefore, just specify CPU indices as though they've been
reserved contiguously (i.e., 0 through 7), and if this is not the case, the
program will automatically map your specified CPU indices to the real indices
that were reserved.
The second argument is the decimal percentage (1-100) of load you'd like to generate on that CPU. It will generate load close to this number. However, if you generate multiple loads on the same CPU index in parallel, it may not do what you'd expect as each of the load generators don't know about each other and will independently assume they have total control over that CPU.
The third and final argument is the number of seconds for which the load should be generated.
The memory module allocates and holds memory for a given duration. The module
has two arguments.
The first argument is the amount of memory to allocate and hold. It is given by
an integer followed by a multiplier suffix. For example, 32MiB means allocate
32*1024*1024=33554432 bytes of memory. The valid suffixes are provided below.
KB = K = KiB = 1024
MB = M = MiB = 1024 * K
GB = G = GiB = 1024 * M
TB = T = TiB = 1024 * G
PB = P = PiB = 1024 * T
EB = E = EiB = 1024 * P
The second and final argument is the number of seconds for which the allocated memory should be held.