Loss returns Nan

[terencenwz]

Loss returns Nan

Some of my settings
causal=true
blindspot_size=1
n_local_attn_heads
ff_chunks=1
reversible=false
use_axial_pos_emb=false

[lucidrains]

Hi Terence! do you want to give the latest version a try?

[terencenwz]

Hi Terence! do you want to give the latest version a try?

Loss goes to Nan after 50k updates for 0.18.1

[ShivanshuPurohit]

How do you train that far? I'm using the deepspeed example and it terminates after 3k steps with seq_len 256, but at least until then the loss doesn't nan.