Add GPU measurements to Benchmark Script

[fabianlim]

We should include the gpu consumption into FILE_RESULTS

• new column gpu_mem (this can be in MiB)

The GPU memory can be collected by:

• starting a async process running nvidia-smi and log to a FILE_MEM
• perform the subprocess.run
• read FILE_MEM and process out how much memory was consumed

Points to note:

• As part of the processing we should perform some estimation based on the gpu memory consumed by the device before the subprocess.run and account for that appropriately.
• If there are multiple GPUs used how should we report? should it be the average?
• We use CUDA_VISIBLE_DEVICES as environment to the subprocess, but Im not sure which GPU nvidia-smi will return. The best is to specifiy the device /s ids to nvidia-smi call.
• we ise nvidia-smi which reports the reserved_memory not the allocated_memory. This needs to be documented.