Async optimizer state and model checkpointing

[chinthysl]

Additional feature to checkpoint optimizer state and model parameters using a non blocking background thread.
Memcopy device buffers to pined host buffer in one shot and let the background thread do I/O operations.

In my 8xA100 setup rough latency improvement is 5.4 sec to 2.3 sec ~ 2X improvement.
When it comes to the larger model sizes this feature will save a lot of time.

[ademeure]

My experience with cudaMallocHost is it can be much slower than a regular malloc which eventually "pays for itself" because the data transfer is so much faster, but for a single transfer, I suspect it might be slower than just a regular malloc.

So if that holds true here, this might need to either keep that memory allocated permanently or just use a regular malloc.