Compute WER metric iteratively

[albertvillanova]

Compute WER metric iteratively to avoid MemoryError.

Fix #2078.

[lhoestq]

LGTM thank you !

cc @patrickvonplaten any opinion on this ?

[elgeish]

Just for safety, you may want to handle the case of total = 0 (e.g., empty set). Unit tests to verify the change would be great as well. Thanks for making this change!

[albertvillanova]

Thanks @elgeish for your comments.

The reasons why I have not explicitly handled the edge case total = 0 are:

• compute_measure does already raise a ValueError exception if any of the references has length = 0: see https://github.com/jitsi/jiwer/blob/f8e5404e4ddb7259081191443def1b6670480244/jiwer/measures.py#L242
• if references and predictions have length = 0, the previous behavior was also raising the same ValueError; the current implementation will raise a ZeroDivisionError in this case

In relation to unit tests, currently we do not test scripts. Maybe @lhoestq can give more insight on this.

[elgeish]

Sounds good; makes sense!

[lhoestq]

I discussed with Patrick and I think we could have a nice addition: have a parameter concatenate_texts that, if True, uses the old implementation.

By default concatenate_texts would be False, so that sentences are evaluated independently, and to save resources (the WER computation has a quadratic complexity).

Some users might still want to use the old implementation.

[albertvillanova]

@lhoestq @patrickvonplaten are you sure of the parameter name concatenate_texts? I was thinking about something like iter...

[lhoestq]

Not sure about the name, if you can improve it feel free to do so ^^'
The old implementation computes the WER on the concatenation of all the input texts, while the new one makes WER measures computation independent for each reference/prediction pair.
That's why I thought of concatenate_texts

[albertvillanova]

@lhoestq yes, but the end user does not necessarily know the details of the implementation of the WER computation.

From the end user perspective I think it might make more sense: how do you want to compute the metric?

• all in once, more RAM memory needed?
• iteratively, less RAM requirements?

Because of that I was thinking of something like iter or iterative...

[lhoestq]

Personally like concatenate_texts better since I feel like iter or iterate are quite vague

[albertvillanova]

Therefore, you can merge... ;)

[lhoestq]

Ok ! merging :)