[Metrics] WER

[justusschock]

Add WER cc @oplatek

Lightning-AI/pytorch-lightning#973 (comment)

[stas6626]

I can work on this issue, but I need help with defining UI, originally this metric works over strings, is it okay if metrics takes list of strings as input? It is possible to run it on tokens, but it will be working right only with char based tokenizer(nowadays most people use BPE, etc)

[stas6626]

@Borda what do you think?

[justusschock]

@stas6626 we already have bleu working on strings. So this wouldn't be a problem. Why do you want to have list of strings here? Shouldn't a predicted and a groundtruth string be sufficient?

[justusschock]

@stas6626 How is it going there?

[VinhLoiIT]

I believe both CER (Character Error Rate) and WER (Word Error Rate) metrics are pretty commonly used in OCR problems and might be in audio/speech as well.

I did implement the metrics in Lightning before but in the older version, so I think it would not be a problem to upgrade the code. However, it might rely on another library such as editdistance that I don't know whether I could be able to use the external library or not when making a contribution.

[janvainer]

It could be interesting to use pytorch-edit-distance instead of editdistance. It should support CUDA operations.

[VinhLoiIT]

I think this implementation has some redundant features, such as remove_blank and strip, which are not necessary, but the CUDA support is useful though.

Moreover, the current implementation requires both blank and space characters, as I skim through the CUDA code, which does not cover the general use of this metric. For example:

• In OCR, we also need the CER (Character Error Rate) metric, which is basically the same as WER but it keeps the space character as a token as well.
• In OCR, splitting words should be based on a user-defined function that should not be coalesence with compute_wer.
• Lastly, in my opinion, the user when using this metric does not have to know about the blank character, because the blank character is only specifically used when we use CTC Loss but sometimes we don't.

I think, in general, both CER and WER metrics are basically computed using the Levenshtein distance whose inputs might only require a list of string of predicted tokens and the list of coresponding target tokens. Thus, editdistance package is just fine though.

[carmocca]

I agree with you @VinhLoiIT

Although I have found the package https://github.com/life4/textdistance to be better supported, for example: roy-ht/editdistance#46

If we agree on a package, we can move forward with the implementation