Profiling and metrics

Overview

Here I will detail the profiling results of EPIC's first Bifrost-based iteration. Briefly, the EPIC code creates a pipeline where the data streams move through multiple processing (or Operating) blocks (e.g., `DecimationOp) via ring buffers. Based on the user-specified settings, the EPIC program may produce images up to a bandwidth of 3.3 MHz at all four (XX, YY, XY, YX) polarizations. Each Op-block is run on a separate thread, bound to its own CPU, while the entire process currently utilizes only one of the two available GPUs on the EPIC node at SV. Below I will only describe results pertaining to the GPU and I will post CPU profiling on a separate thread.

Before I show the results, I will describe the tools used. I used NVIDIA Nsight Systems to measure the overall performance (or system-level profiling for the GPU-peeps), and NVIDIA Nsight Compute to measure the kernel metrics. See Pearson 2020 for a nice overview of these tools. For both tools, I ran EPIC with the following options to profile the code with a total runtime of 50 s:

• Gulp size 40 ms (or --nts 1000)
• Single polarization
• Image size 64x64
• Image resolution 1.8 deg
• Accumulation time (80 ms)
• Channels 90

Run command:

LD_PRELOAD="libvma.so" VMA_RX_POLL=1000 VMA_INTERNAL_THREAD_AFFINITY=0 VMA_RX_PREFETCH_BYTES=128\
 VMA_THREAD_MODE=0 VMA_MTU=9000 VMA_TRACELEVEL=0 numactl --cpubind=1 --membind=1\
 ./LWA_bifrost.py  --addr 239.168.40.16 --port 4015 --channels 90\
 --imagesize 64 --imageres 1.8  --accumulate 80 --nts 1000 --singlepol --duration 50

Results

References

• Carl Pearson 2020, Using Nsight Compute and Nsight Systems