Changes
- Set up EMA/SWA in training. Played with EMA a bit and found a low coefficient like 0.8 is needed for eval loss to converge well. Did not see eval improvement yet over not using EMA
- Tune the max period in time sinusoidal embedding and RoPE. Default to 100.0 and 1000.0 for now (from 10000.0 before)
- Skip post-attention operations in the last layer with proprio (not needed, like in paligemma)
- Add strict package versions for bitsandbytes and transformers
- Clean up installation instructions and training logging
Other notes
- Try using weight decay as high as 10 for both action expert and paligemma