This is a simple mindscript tokenizer to make your chatbot better understand user questions.
- Install the dependencies:
npm install @mindlearn/tokenizer- Use the tokenizer in your code:
const tokenizer = require("@mindlearn/tokenizer");
const word = "Hello World, here is a basic mindscript tokenizer example.";
console.log(tokenizer(word));