NLU clean up data set

[secretsauceai]

Description

We want to be able to benchmark NLU data sets, then refine the entries of the data sets. This is done to improve the quality of the data set.
Please see cleaning the NLU dataset and macro NLU data refinement.

High-level deliverables

• Creating basic data cleaner (formatting, etc.)
• Build NLU engine for testing
  • intent classifier
  • entity classifier
  • testing pipeline
• Create flows for refining the NLU data
  • Intent refinement flow (almost done!)
  • Entity refinement flow (will hopefully be similar to above!)
• Refine data set

DoD

Related milestones

The related milestones are in NLU Engine Prototype Benchmarks repo.

• Create refined intent data
• Create entity refinement data set