feat: add string translating/encoding utilities #116
Conversation
Codecov Report
@@ Coverage Diff @@
## main #116 +/- ##
==========================================
- Coverage 97.53% 97.41% -0.12%
==========================================
Files 29 32 +3
Lines 972 1005 +33
==========================================
+ Hits 948 979 +31
- Misses 24 26 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
doctr/datasets/utils.py
Outdated
| char = unicodedata.normalize('NFD', char).encode('ascii', 'ignore').decode('ascii') | ||
| if char == '' or char not in vocabs[vocab]: | ||
| # if normalization fails or char still not in vocab, return a black square (unknown symbol) | ||
| char = '■' |
There was a problem hiding this comment.
Mmmh, don't you think it would be safer for us to hardcode the conversion between vocabs?
There was a problem hiding this comment.
Suppose you have a dataset with such a word: ¢¾©téØßřůëðÛ@læ5€ë𮵶
You need to map each character to our vocab, I agree I would be perfect to control it manually, but you have hundreds of different characters (russian, greek, math, symbols, punctuation, shapes, norvegian, chineese, ...) and for each of these char you have different possible matching for each vocab so the attribution task seems hardly achievable manually.
This PR is linked to issue #115, it implements:
A dictionnary of vocabularies is also provided
Any feeback is welcome!