Metrics to be included (proposal)

[bact]

Metrics to be included and other discussions, moved over from PyThaiNLP/pythainlp#62

Quality

• Does the tokenizer preserved every characters or does it destructive?
• Character-level
  • If it violates Thai Character Cluster rule? (more penalty)
• Word-level
  • Word-level by length of word

Speed

• Characters per second (on standardized machine)
  • May be tested with different sizes of text (small and large), to notice the "boot time" of a tokenizer

Memory footprint

• Memory used by the tokenizer (when tokenizing a certain amount of text), at the running time

Disk size

• Total size of the tokenizer, including dictionary, models, and all non-standard dependencies (excluding runtime environment, like interpreter/VM)

[p16i]

@bact thanks for the proposal.

Because this repository is mainly for comparing end results, i.e. tokenised texts, only quality metrics can be measured.

However, speed, memory and disk footprint have to be done while executing the algorithms. I think we can measure them via https://github.com/heytitle/thai-tokenisers.

I'm going to close this issue and we will discuss further details in #9 and PyThaiNLP/docker-thai-tokenizers#1.

@bact I've also put some thoughts on how to implement those performance metrics. Could you please comment on my idea?

[bact]

Thanks krub, will see it over there.