Models for experiments

[snakers4]

Models envisioned:

• TCN / LSTM / GRU as sequential models;
• Decided to abandon HLSTM, indie LSTM, transformer for various reasons;
• Attention schemes:
  • Average pooling
  • Plain self-attention
  • Multi-head self-attention with multiple classification heads
• Tokenization schemes
  • Dictionary (and / or BPE)
  • Embedding bag
  • Embedding bag with attention
  • CNN + char level

[snakers4]

It seems that I have prepared all of the major models