-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Custom words dictionary #73
Comments
Hi @venzen I can confirm that deepcut cannot do the text correction or spelling correction. You may have to create a function (after OCR) to deal with these corrections before performing the tokenization. For the second question, I think deepcut should generalized enough to parse new unseen words. However, you can put list of words in the dictionary to cover some cases that you think that it may tokenize wrongly. Hope this help a bit! Maybe someone can follow this issue too! |
There was a paper for spelling correction by Ekapol and team which may help you quite a bit: https://ieeexplore.ieee.org/document/9145483. I'm not sure if they provide open-source implementation somewhere. |
@titipata Thank you for your response. I started implementing word similarity checking before I saw your reply. Can use As you recommend, I then add new words to this file and add it as a Thank you, also, for sending the link to the paper about Thai spelling correction. I will read and give feedback if I find a solution. |
Unexpected behavior from The string is:
['หรืออิริยาบถ', 'น้อย', ' ', 'มี', ... ] Any idea what is happening in this case? EDIT: I should add that my custom dictionary includes 19,000 Thai words. Some of them are compound words. Perhaps this is causing strange behavior. |
The issue was that the custom dictionary contained duplicate words (words also present in the deepcut dictionary). When I made a new blank custom dictionary deepcut works as expected. |
Thank you for this good work. I have two questions about using this tool. First let me briefly explain my use case:
I am translating Buddhist texts from Thai to English for the Mahachulalangkornraachawitayaalay (MCU). The source material is images, so I must first do OCR (with
tesseract
) and then edit to markdown format. After that I can translate to English using Google Translate. During OCR some characters and annotations are missed or misinterpreted. I hope thatdeepcut
can allow me to correct those words that are misrepresented by OCR. For example, the correct word is 'ประจําบท' but OCR misses the sara am and returns 'ประจาบท'.deepcut
help in this case?deepcut
for identification in the future?The text was updated successfully, but these errors were encountered: