Skip to content

Source codes and materials of Advanced Spelling Error Correction project.

License

Notifications You must be signed in to change notification settings

Amir79Naziri/SpellCorrectionML

Repository files navigation

Advanced Spelling Error Correction in Persian Using BERT: A Comprehensive Approach

Writing, a pervasive form of human communication, permeates nearly every facet of contemporary life. Consequently, any inaccuracies or errors in written communication can have profound consequences, from financial losses to potentially life-threatening situations. Spelling mistakes, one of the most common writing errors, are encountered frequently due to various factors. This research aims to identify and rectify diverse spelling errors in text by harnessing neural networks, particularly the Bidirectional Encoder Representations from Transformers (BERT) masked language model. To achieve this, a comprehensive dataset encompassing both non-real word and real-word errors was compiled after categorizing different types of spelling mistakes. Multiple pre-trained ParsBERT models were subsequently employed.

To ensure optimal performance in correcting misspelling errors, we proposed a combined approach utilizing the BERT masked language model and Levenshtein distance. The results from our evaluation data demonstrate that the system presented herein exhibits remarkable capabilities in identifying and rectifying spelling mistakes, often outperforming existing systems for the Persian language.

Keywords

Spelling mistakes, Neural Networks, BERT masked language model, Error correction system, Real and non-real word errors, ParsBERT model, Levenshtein distance