error: input contains invalid UTF-8 around byte 30 (of 68)

[talalriaz]

I encountered this error while running the following code:

import pycld2 as cld2
text ="""
Happy Tailors Day! Hackett We�re celebrating with a special offer
"""
isReliable, textBytesFound, details =  cld2.detect(text)

Here is the error:

error: input contains invalid UTF-8 around byte 30 (of 68)

[ned2]

There's been some great exploration of this issue in this polyglot issue and [also in the older cld2 project(https://github.com/mikemccand/chromium-compact-language-detector/issues/22) that pycld2 is forked from (some of which are from folks using Polyglot, which actually depends on pycld2 rather than that older cld2 project).

Have not tried it yet, but this solution, which uses a regex to strip the two offending UTF8 control characters from the input, looks like the most elegant solution to me.