Question: Running on local hardware

[getnamo]

What modifications would be required to run this on local hardware?

Can batching be shrunk down to fit commodity hardware such as a 24GB VRam GPU? If linearly scaled, 26x less VRam should turn a 4 hr run into 106hrs, but at <1/4 cost (assuming electricity only vs cloud cost).

[aifartist]

@getnamo
I'm trying to get this to work now on my local threadripper 7985WX, 256GB's of Ram and dual 5090 Ubuntu home system.
There is some bug with:
tokenizer = rustbpe.Tokenizer()
^^^^^^^^^^^^^^^^^
AttributeError: module 'rustbpe' has no attribute 'Tokenizer'

Note that I did get nanoGPT to actually work for both the 'baby' model and the full GPT2-small model although I terminated the later because my estimate on a single 5090 was 17 days. Note that I made a change to the torch.compile in nanoGPT and got a 5X performance increase allowing me to train 'baby' quickly. However, there is a weird problem if I try to do with on the full GPT2-small. If I could fix this I might get this 17 Days to about 3.

However, as of today I've jumped on nanochat.

[StrongerXi]

Note that I made a change to the torch.compile in nanoGPT and got a 5X performance increase allowing me to train 'baby' quickly

What is the change? @aifartist

[aifartist]

Note that I made a change to the torch.compile in nanoGPT and got a 5X performance increase allowing me to train 'baby' quickly

What is the change? @aifartist

I did get nanochat to work on my dual 5090's but I had to shutdown all my chrome browser windows to let it use all the GPU VRam. I also had to reduce the batch size from 32 to 8. This is why I've been offline.

I let it run long enough to estimate that if I want to let it complete it would take about 27.8 hours to complete all the training steps. A slight reduction from the 29.7 hours before I made some perf tweaks.

However, back to the big 5X thing. It was using mode='max-autotune' for torch.compile(). The problem is that it isn't working for nanoGPT GPT2 nor nanochat. I was able to get mode='max-autotune-no-cudagraphs' to work which probably gave me the small speed up. However, getting a full cudagraph with no breaks might achieve a quite large speed up.

I'm debugging the max autotune problem now, but hmmm, I can probably leave my browsers running if I just set the batchsize down to 4 until I figure out the problem. I just need to keep instrumenting and restarting the run until I find the root cause.

[amitkoth]

(Random aside) out of curiosity - I had a look at comparing it at a planning/cursory level to the newly announced DGX Spark machine. Obviously that machine is really for inference rather than training.

---

For GPT-2 Scale (1.5B parameters):

8xH100: ~12 hours (per nanochat's d26 model)
DGX Spark: ~960-1,200 hours (40-50 days)

For GPT-3 Scale (175B parameters):

8xH100: Requires multiple nodes, ~1,000 GPU-days
DGX Spark: Physically impossible - exceeds memory capacity

[aifartist]

@mohanjangir Definitely feasible on my dual 5090 setup given I have gotten the training to run for a bit. Seems to be efficiently running. It wouldn't take a huge amount of time but I decided to kill the run as I'm been a perf freak for 45+ years and now doing a deep dive into torch.compile and Triton.

If I can make this twice as fast or faster I'll make a name for myself. :-)

The key based on prior experience is finding if I can leverage CUDA graphs even partially. I am literally studying the execution graph creation during tracing of the model() call which is later used in the loss.backwards() call.
I have a crazy idea how to fix this and my fingers are crossed.

[shizhediao]

I encountered the same issue:

tokenizer = rustbpe.Tokenizer()
^^^^^^^^^^^^^^^^^
AttributeError: module 'rustbpe' has no attribute 'Tokenizer'

[aifartist]

I encountered the same issue:

tokenizer = rustbpe.Tokenizer()
^^^^^^^^^^^^^^^^^
AttributeError: module 'rustbpe' has no attribute 'Tokenizer'

I had eventually figured this out. I missed one of the setup steps. I tried to manually do everything and missed one of the things speedrun.sh was doing. Did you use it and did you watch for errors on all the steps.

[shizhediao]

Indeed. I tried to solve all errors and warnings and it works. Potentials issues are conda env (I used it before and need to unset it by unset CONDA_PREFIX) and figure out the path of uv.

[JonathanFly]

What modifications would be required to run this on local hardware?

Can batching be shrunk down to fit commodity hardware such as a 24GB VRam GPU? If linearly scaled, 26x less VRam should turn a 4 hr run into 106hrs, but at <1/4 cost (assuming electricity only vs cloud cost).

You can save a little memory on a single GPU by skipping torchrun. For example:

python -m scripts.base_train --depth=20 --device_batch_size=4 --sample_every=100

It's ~17 gig vs ~21 with torchrun. As far I know you don't lose anything significant skipping torchrun with a single GPU. (You don't need that sample_every=100, but it is fun to see some outputs without waiting for hours like the defaults)