Detected encoding is wrong with DetectFromBytes, ok with other methods for UTF-8 file containing emoji

[fretman92]

Test program launched from latest source:

            string filename = args[0];

            var result = CharsetDetector.DetectFromFile(filename);

            if (result.Detected != null)
            {
                Console.WriteLine("DetectFromFile - Charset: {0}, confidence: {1}", result.Detected.EncodingName, result.Detected.Confidence);
            }

            byte[] bytes = System.IO.File.ReadAllBytes(filename);
            result = CharsetDetector.DetectFromBytes(bytes);

            if (result.Detected != null)
            {
                Console.WriteLine("DetectFromBytes - Charset: {0}, confidence: {1}", result.Detected.EncodingName, result.Detected.Confidence);
            }

            System.IO.Stream fileStream = System.IO.File.OpenRead(filename);
            result = CharsetDetector.DetectFromStream(fileStream);

            if (result.Detected != null)
            {
                Console.WriteLine("DetectFromStream - Charset: {0}, confidence: {1}", result.Detected.EncodingName, result.Detected.Confidence);
            }

Result:

The file is a HTML UTF-8 (without BOM) encoded file containing 1 simple emoji : 😀
(attached in the zip below)
utf8_with_emoji.zip

Why does the DetectFromBytes method gives a different result?

[304NotModified]

Why does the DetectFromBytes method gives a different result?

Need some time, but figured something out

• The detect stream read in buffers of 1024 bytes. After 3 cycles, it's convinced it's UTF-8
• The detect bytes reads all bytes (as they al already in memory) (73939 bytes), and after that It don't think it's ut8
  --

Now we have to check why it's after 3072 bytes UTF-8 and not after 73939 bytes.

[304NotModified]

it failes at first at bytecount 73837 (so last 102 bytes)

[304NotModified]

F0 9F 98 80 is not recognized as utf8 :(

[rstm-sf]

In general, emoji is another logic :)

[rstm-sf]

Maybe can come up with a table of special characters that it need to skip in a certain scenario

[304NotModified]

replaced with #149