hPong (WIP)

Doing:

• implement temporal position embedding using rope

Todo:

• Read up on rotary position embeddings in transformers, look at example implementations
• implement spacial position embedding using rope
• make vae training output images after each epoch
• Figure out why KL divergence of the DiT is so high (in the 9 orders of magnitude)
• guidance for user actions on DiT (STGuidance)

Done:

• Split each frame into its own VAE encoding/decoding.
• in the dataset, the "player" paddle should sometimes play like a good player for data quality

References

• RoPE: Rotary Position Embedding
• Spatiotemporal Skip Guidance
• Classifier-free Guidance
• Axial Attention