Hardware used for reproducing

[nathan-az]

Any information on the exact hardware used when training Zephyr 7B Beta? The deepspeed config looks like no CPU offloading was done so were larger (or many) GPUs used?

Would be great to know the hardware that was used for the sake of reproducing, as well as safely expanding to using other datasets (knowing that the current configuration with regards to batch sizes won't OOM).

[edbeeching]

I believe we used 2 nodes with 8xA100 80GB per node. But you should be able to trained it on 1 node if you adjust the batch size and grad accumulation.