Meeting 12.10.2021

Discussions

• Review of the labels and our classification approach
• Standard vs Parallel MLP vs hierarchy MLP
  • Action: prioritize standard setup, but test others
• Relation to deployment constraints
  • runtime, memory, and latency constraints
  • Action: test in a realistic setting with Lambda (delay, concurrent included)
  • Action: determine run-time constraints (length of text)
• Optimizing dataset, training procedure? (->Abdullah?)
• Error analysis
  • Per-framework performance (question: merge or split datasets?)
  • Action: train with different AFs and use the same test set
• Threshold analysis
  • Biased estimate using train/test split (good for now)
• Evaluation
  • Curation of train/test set: stratified split
  • Respect the ratio of negative samples in the test
  • Training set: use any ratio best for training
  • Action: prepare the new evaluation sets
  • Action: re-do any prior analysis
• Number-dependent tags
  • Data augmentation by changing the numbers
  • Action: change numbers and dates to meta-tags (special token)
  • Action: simple NER for numbers and dates