Performance Profile

When the pipeline is ran with -P parameter performance metrics including the CPU, GPU and memory usage will be recorded during processing. This information is then collected and displayed in a Jupyter notebook. This requires pyhton, jupyter, and if using GPU CUDA acceleration, nvidia-smi. Performance metrics are captured using the following applications: top, vmstat, iostat, nfsiostat and nvidia-smi. Output from these are captured and stored under a logs directory inside the ExSeq directory. After the pipeline finishes running with the -P option, python scripts under tests/perf-profile/ aggregate the generate logs into csv files which are then read by the python notebook created perf-measurment.ipynb. Jupyter when run, will open the system default web browser with a new tab with the python notebook which displays plots of the capture data.

Examples

Running Jupyter on a Remote Server

One issue that can arise when running the pipeline on a remote server is Jupyter will by default open a web browser on the remote machine. Depending on the distance to server interactions with the user interface can be slow. To alleviate this it is possible to run the browser locally and the jupyter server remotely. To do so start the jupyter server on the remote system with: jupyter notebook --no-browser --port=8887 The server will start in that terminal window without opening a browser and giving a link like http://localhost:8887/?token=4a8c84dbb27d5ba54c9f1c126a20f8d9a6fc69e31ba4461a. Then on the local machine open up a terminal and run: ssh -N -L localhost:8887:localhost:8887 user@remoteMachine replacing the last parameter with relevant user / machine. This forwards connections to the local port to the remote port and allows the earlier link to be opened in a locally running web browser. This allows for easier viewing of the notebook.