You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@gitlostcomment in #334 about guessing language made me rethink my dumb approach mentioned there and first invest a little in a web search, which resulted in this: https://github.com/google/compact_enc_det. It has been suggested here and seems to be used inside google chrome. It has a compatible license, so I gave it a 'quick' try and hacked it into my build and it does indeed detect the correct encoding of the mentioned upstream sample. It breaks 2 unit tests though, because it guesses something different than the old approach (GB2312 instead of ShiftJIS). The same problem would likely happen with any other self-made 'improvement' or use of any other 3rd party lib.
One annoying problem is that the provided cmake scripts are uncooperative in the FetchContent context and need patches for it to build, so I would need to fork it.
So this looks promising but would likely introduce regressions for end users (while also likely fix issues...).
The text was updated successfully, but these errors were encountered:
@gitlost comment in #334 about guessing language made me rethink my dumb approach mentioned there and first invest a little in a web search, which resulted in this: https://github.com/google/compact_enc_det. It has been suggested here and seems to be used inside google chrome. It has a compatible license, so I gave it a 'quick' try and hacked it into my build and it does indeed detect the correct encoding of the mentioned upstream sample. It breaks 2 unit tests though, because it guesses something different than the old approach (
GB2312
instead ofShiftJIS
). The same problem would likely happen with any other self-made 'improvement' or use of any other 3rd party lib.One annoying problem is that the provided cmake scripts are uncooperative in the
FetchContent
context and need patches for it to build, so I would need to fork it.So this looks promising but would likely introduce regressions for end users (while also likely fix issues...).
The text was updated successfully, but these errors were encountered: