You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have encountered an error when using the GitHub cli to fetch commits in an MDN repository. And I found this error is coursed by the sanitizer which is used by GitHub cli.
When we read the plain text, and use transform with the the sanitizer , we would got an error:
But this should be the correct text. I found the error is returned here.
So I read the signature of utf8.DecodeRune. It may also return utf8.RuneError if the bytes are correctly decoded. And if there does be a decode error, it will return (RuneError, 0) or (RuneError, 1).
So we can't judge whether there is a decoding error just based on the first value returned, like the text I used above, which uses this unicode character. The sanitizer mishandled it.
The text was updated successfully, but these errors were encountered:
@yin1999 Thanks for raising this issue. I was able to reproduce it, the sanitizer is in fact mishandling properly encoded \uFFFD unicode characters that are coming from the API.
We have encountered an error when using the GitHub
cli
to fetch commits in an MDN repository. And I found this error is coursed by the sanitizer which is used by GitHubcli
.So I created a demo to reproduce the problem:
the plain text to transform:
When we read the plain text, and use
transform
with the the sanitizer , we would got an error:But this should be the correct text. I found the error is returned here.
So I read the signature of
utf8.DecodeRune
. It may also returnutf8.RuneError
if the bytes are correctly decoded. And if there does be a decode error, it will return(RuneError, 0)
or(RuneError, 1)
.So we can't judge whether there is a decoding error just based on the first value returned, like the text I used above, which uses this unicode character. The sanitizer mishandled it.
The text was updated successfully, but these errors were encountered: