Running Spatter

Last updated: March 30th, 2019

While we eventually plan to script these selections, it is useful to understand how Spatter allocates buffer space to know the max data size and sparsity you can use on a specific system.

Let's say we want to do a gather operation that transfers 500 MB with sparsity 1/2 or every two elements:

1. Each element is a double or 8 bytes.
2. To perform a gather, we need a wide buffer (2x the size of the gathered amount of data) for the source buffer, an index buffer that is 1x the size of the data to be moved and a target buffer that is 1x the data size to be moved. This means we would need 4x the size of the number of bytes moved, or 2 GB of RAM to perform a 500 MB gather.

For transferring 500 MB measured with powers of 10 (as opposed to MiB), that would be (500*10^6 B) / (8 B / element) = 62,500,000 elements transferred.

So we'd run this test as follows:

#Requires 2 GB of RAM to avoid paging to disk
#10 iterations are run by default - this can be changed with the -runs flag
./spatter -s 2 -l 62500000 -runs 4
Warning: Length not specified. Default is 16 (elements)
Warning: No backend specified, guessing OpenMP
Warning: Kernel unspecified, guess GATHER
Warning: Kernel file unspecified, guessing kernels/kernels_vector.cl
backend kernel op time source_size target_size idx_size bytes_moved usable_bandwidth actual_bandwidth omp_threads vector_len block_dim
OPENMP GATHER COPY 0.000054 256 128 128 256 4.784600 7.176899 128 1 1 0
OPENMP GATHER COPY 0.000117 256 128 128 256 2.196275 3.294412 128 1 1 0
OPENMP GATHER COPY 0.000086 256 128 128 256 2.977714 4.466571 128 1 1 0
OPENMP GATHER COPY 0.000078 256 128 128 256 3.288206 4.932309 128 1 1 0

Larger test cases and using the wrapping parameter

Spatter can also use wrapping to perform larger scatter/gather operations. This means that the benchmark uses different different elements when going over the source buffer again, meaning the second pass over the indices may have worse alignment than the first. Wrapping is not turned on by default but can be selected for larger scatter/gather operations using the -w flag with a value less than or equal to the selected sparsity.

Let's test the case of transferring 500 GB of data (8388608000 * 8 / (10^9)) = 67GB data moved. Total memory required would be 67 GB for the Index buffer, 67*2GB for the Source buffer, and 67 GB for the Target buffer, which is about 268 GB.

./spatter -s [sparsity] -l [num_doubles_moved] -w [wrap must be <= sparsity]
#To test with sparsity 1/2 on 500 GB gather with wrapping factor of 2
`./spatter -s 2 -l 8388608000 -w 2`
Warning: No backend specified, guessing OpenMP
Warning: Kernel unspecified, guess GATHER
Warning: Kernel file unspecified, guessing kernels/kernels_vector.cl
backend kernel op time source_size target_size idx_size bytes_moved usable_bandwidth actual_bandwidth omp_threads vector_len block_dim
OPENMP GATHER COPY 23.080705 134217728000 67108864000 67108864000 134217728000 5815.148492 8722.722738 128 1 1 0
OPENMP GATHER COPY 5.232357 134217728000 67108864000 67108864000 134217728000 25651.485157 38477.227736 128 1 1 0