sequence_weirdness returns -1 with U+2019 ’ [Pf] RIGHT SINGLE QUOTATION MARK

[guillaumeldc]

ftfy.badness.sequence_weirdness("Don’t Follow the Leader")
-1

is this expected?

I looked into the documentation but having a hard time interpreting this result. fix_text works as expected and correct the quote which solves my issue but I wanted to flag my bad strings using a (> 0) condition like in one of your example.
is it safe to just use != 0 instead?

[rspeer]

Perhaps sequence_weirdness should have had a leading underscore, except I ended up using it in tests. It's a metric used internally for deciding if mojibake should be, or has been, fixed. The "sequence weirdness" of a curly quote is negative because it is a character that could appear in mojibake, but actually is much more likely to occur in the real, intended text.

sequence_weirdness is unrelated to the fixer that uncurls quotes, for example.

I don't see any real purpose in asking if sequence_weirdness is positive or non-zero.

[kretes]

Hello.

First of all - thanks for a great tool!

Commenting on negative result of sequence_weirdness - this is assumed to be a way of checking the text before taking a decision on whether to investigate deeper/fix.
And this is how it is explained in the docs: Heuristics to determine whether re-encoding text is actually making it more reasonable.

So - I assume that the logic 'look just at text with weirdness > 0' is normal one. And from my limited experience it looks like a good method.

Returning -1 for means that if some mojibake is in the text together with some of this characters that give -1 - the overall value is zero again:

import ftfy
ftfy.badness.sequence_weirdness("п®")
0
ftfy.badness.sequence_weirdness("п")
1
ftfy.badness.sequence_weirdness("®")
-1

and it makes heuristic for > 0 less accurate