Add optional character-level metrics (CER / cpCER / tcpCER) for languages with ambiguous word segmentation

[hwanython]

Hi,

Thanks for the great toolkit. I've been using MeetEval recently while experimenting with multi-speaker ASR evaluation.

I’m an engineer/researcher from Korea, working with multilingual speech recognition systems.

While using MeetEval, I noticed that the current metrics are mainly WER-based (WER, cpWER, tcpWER). This works well for languages with clear whitespace word boundaries like English.

However for languages such as:

• Chinese
• Japanese
• Korean

word segmentation can be ambiguous, and evaluation scores can depend on the tokenizer used. Because of this, many ASR benchmarks for these languages also report Character Error Rate (CER).

So I was wondering whether it would make sense to add character-level variants of the existing metrics, for example:

• CER
• cpCER
• tcpCER

Conceptually this would only change the evaluation unit (characters instead of words) while keeping the existing pipeline (speaker permutation, timing constraints, etc.) the same.

Before trying to implement this, I wanted to ask:

• Would adding character-level metrics fit within the scope of MeetEval?
• If yes, would you prefer a separate CER implementation, or a more general token unit abstraction?

If this sounds useful, I’d be happy to try preparing a PR.

Thanks!

[thequilo]

Hi @hwanython!

We were not sure how to implement this and for now decided to let the user split into characters/tokens/words by inserting spaces into the transcript. Then the WER function return a CER. The reason for this decision is that we didn't want to implement something that in the end doesn't match the actual requirements for the non-latin/non-word-based languages. Letting the user do the split gives them most flexibility and keeps us from implementing something wrongly.

Related is also #116 for estimating pronunciation lengths for character-based languages. For reference, here's what we have decided on in that PR: #116 (comment).

That being said, I would be happy to accept an implementation of either a word-level split script or a CER api at meeteval.cer that performs the split and then calls the corresponding functions in meeteval.wer. Or both (also for better testing).

[hwanython]

Thanks for the clarification!

That approach makes sense. Keeping the core WER implementation unchanged and implementing CER as a wrapper sounds like a good design.

My understanding is that the implementation could look like:

meeteval.cer
→ split characters
→ call meeteval.wer / cpwer / tcpwer internally

Possibly together with a small helper script for character splitting.

I’ll take a look at the codebase and try preparing a small PR with:

• a simple CER wrapper API (cer / cpcer / tcpcer)
• an optional character split helper
• some basic tests

Since I mainly work with Korean ASR, I’ll also check the behavior with a few Korean examples to make sure the character-level evaluation behaves as expected.

Let me know if you have a preferred structure for this.