Fix fineweb's parallel processing on windows

[ltogniolli]

Fineweb importing fails on Windows with "Attempt to start a new process before the current process has finished its bootstrapping phase." due to platform semantics. Fix it by adding an entry point guard as indicated in the documentation

[lukasugar]

Interestingly, I got the same error on macOS. Your fix worked!

[gerardaristizabalpla4]

THANK YOU!!

Were you able to run everything that Andrej run in Windows? I was having issues with compile and scaled_dot_product_attention because the package Triton apparently can't be downloaded for Windows. Thank You!

[GrahamTheCoder]

@lukasugar Does it run? Yes. Except it isn't as fast because there's no windows implementation of:

• NCCL - so limited to 1 GPU
• Flash attention v2 - so attention is slower
• Triton - so torch.compile isn't available, so again, slower

On a 3090 24GB I get about 52K t/s, if I run it under ubuntu even under WSL2 I get 62K t/s. Baremetal linux was maybe 0.5% faster, but opens up multi-gpu as an option.

For multi-gpu on windows I did try using torch.distributed.init_process_group(backend="gloo") and omitting --standalone from torchrun command, but I get "Cannot use ReduceOp.AVG with Gloo" - it only works with NCCL. I haven't tried treating them as if they are each on separate machines