Stricter FP32 tests

[gordicaleksa]

• Stricter FP32 logit accuracy
• Much stricter FP32 loss accuracy
• Much stricter FP32 grad tensor accuracy (and somewhat stricter 16 bit accuracy)
• Copied over new expected loss values from PyTorch (they're a bit different and I took all 6 decimal points and not just 4)
• Also adapted our test logic to round loss to 6 decimal points

Regarding grad tensors: back when Andrej hardcoded the thresholds we had a bug in PyTorch that led to a bigger discrepancy between our PT vs C code - now that that's fixed we can be really strict and use 1e-6f here.

[ngc92]

can we turn this entire thing into a loop?
have a small struct {size, name, threshold}, which also would make it easier to match thresholds and tensor names?

[gordicaleksa]

agree we could do that but i'd do it in a separate PR

[ngc92]

ideally, we'd export these from pytorch when we generate the reference file instead of having to copy them manually

[gordicaleksa]

hopefully it doesn't change that often? since we're not touching the reference imp that frequently - but if it does then it's worth investing into that

[rosslwheeler]

@gordicaleksa - this appears to be causing a failure when running:

make testgpt2_cu USE_CUDNN=1 && ./testgpt2_cu

It may or may not be seen on your environment but am able to see it here.

@karpathy - I think this is what was causing the issue. I ran my tests one commit back from this in your repo and it passes consistently. If I checkout this commit, then the failures start. Not 100 percent sure but it does make some sense since this is the test that's failing and there aren't many other changes to this file recently?

Can you confirm since you were seeing the failure consistently too? Thank you.

[gordicaleksa]

It's certainly this PR - sad our CI didn't catch this! See #615 for a fix.