Redo the modeling approach and training pipeline

[khanhvu207]

Ideas for the new modeling approach

• 1. Learn with raw fleets' plans altogether (end-to-end learning).
• 2. Predict action: SPAWN, LAUNCH, IDLE.
• 3. Predict the cardinality.
  • Two different types of cardinality with SPAWN and LAUNCH.
  • Multi-class prediction for Spawn_nr.
• 6. Convert the global information such as team kore and opponent kore into a vector?
• 7. Convert the current step in a vector embedding -> Stronger signal I guess?
• 9. Pixel Random Sampling (see Pixel-BERT).
  • Replace some tokens with a learnable token.
  • Or, replace them with a zero vector.
• 10. Be mindful about when to use explicit embedding vectors.
  • Do we need to have SEP token in the src_seq if we do not use positional embeddings?
• 12. bias=True for output heads?
  • I think it is necessary (discussed with Kien about this last week).
• 13. layer_norm for all input tokens.
  • I can try with and w/o layer_norm for input tokens.
• 14. Consider removing the FlightPlanEncoder with an MLP. (something like BoW).
  • Embedding -> MLP -> Sum.
• Compute token accuracy only for LAUNCH action.
• Compute precision/recall for each class SPAWN, LAUNCH, IDLE.

Notes

1. Perhaps consider removing the convolutional layers.
2. Consider using fewer training epochs because the dataset is small?
3. Increase network capacity.