Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Catalan Speech Recognition Benchmark

There are not many speech recognition engines/models for the Catalan language. This benchmark aims to collect and compare the available models and serve as reference for projects willing to include catalan speech recognition.

Table of Contents

Test data

Two datasets have been initially used to benchmark the different models:

As it is not possible to assure that neither of the test datasets has been used when training the evaluated models, a new dataset is in preparation for future tests.


This benchmark considers three metrics: word error rate, real-time factor and model size.

Word error rate (WER)

Defined as the ratio of the Levenshtein distance between the reference transcript and the prediction, to the number of words in the reference transcript.

Real-time factor (RTF)

The ratio of the recognition response time to the audio length. The smaller the ratio, the faster it takes to infer speech. This is not considered for cloud-based recognition engines.

As this metric depends on the hardware the model runs, all tests have been executed sequentially on the same machine, an AWS g4dn.xlarge instance:

  • 4 vCPU (2nd Generation Intel Xeon Scalable (Cascade Lake) processors)
  • 16 GiB Memory
  • NVIDIA T4 Tensor Core GPU (16 GiB GPU Mem)

GPU was used when the model supported it.

Model size

The uncompressed size of the model as needed by the benchmarking scripts.

Models / Engines

The following models have been tested, if you publish or know of any other speech recognition model for the Catalan language please open an issue!

Col·lectivaT CMUSphinx v0.4.0

A CMUSphinx based model released by Col·lectivaT in November 2018 with AGPL-3.0 license. The model was trained on the ParlamentParla and TV3Parla datasets. More information can be found in their website or github repository.

deepspeech-català v0.7.0

A Mozilla DeepSpeech based speech recognition model, trained on the Common Voice June dataset, ParlamentParla and FestCat.


A Vosk-API compatible lightweight wideband model for Android and RPi for Catalan by Alpha Cephei

Google Speech-To-Text

The speech recognition service offered by Google

Azure Cognitive Services

The speech to text service offered by Azure


Crowdsourced google dataset

Model/Engine WER RTF Model size
Col·lectivaT CMUSphinx v0.4.0 45,89% 0,69 106 MB
deepspeech-català v0.7.0 21,69% 0,18 233 MB
vosk-model-small-ca-0.4 11,46% 0,04 91 MB
Google Speech-To-Text 10,97% N/A N/A
Azure Cognitive Services 11,72% N/A N/A

Common Voice June 2020 Test dataset

Model/Engine WER RTF Model size
Col·lectivaT CMUSphinx v0.4.0 TBD TBD 106 MB
deepspeech-català v0.7.0 18,41% 0,18 233 MB
vosk-model-small-ca-0.4 11,71% 0,04 91 MB
Google Speech-To-Text TBD N/A N/A
Azure Cognitive Services TBD N/A N/A


  • Both Google and Azure convert numbers and time to their numerical representation (quite accurately) but the original test data represents them textually. These were converted back in a semi-automatic post-processing step in order to correct the WER when they were semantically correct.
  • Azure also adds punctuation, which was removed in the post-processing step.
  • Azure fails to add middle dots and hyphens, which worsens its WER notably. These were not corrected as it didn't seem right to do so (the previous two corrections did not change the semantics of the sentences whilst failing to add hyphens or middle dots renders different words)
  • Azure's profanity check appears not to be language specific as it marked "munt" (mount, hill) and "retard" (delay) as offensive.