8x RTX Pro 6000 S GPUs pre-trained in ~2hr 15m

[mdhorton]

My local setup has 2x RTX Pro 4000 blackwell GPUs and I wanted to see how nanochat pre-training performed locally. Out of curiosity I ported nanochat to c++ and just kept going.

The main target is sm120 (non-datacenter blackwells). So this is not something generic to be ported back to H100s.

On rented 8x RTX Pro 6000 S GPUs it finished in ~2hr 15mins.

Here's a brief summary:

GPU	GPUs	device batch size	bpb	CORE	time
RTX Pro 6000 S	8	8	0.71773	0.2592	135.07m
RTX 5090	8	4	0.71769	0.2615	148.85m
RTX Pro 4000	2	2	0.71934	0.2616	1622.59m

More details in the readme:
https://github.com/mdhorton/nanochat-cpp