chardet 3.0.0 breaks the behavior of 2.3.0 without a depreciation notice.

[luzfcb]

hello, I was trying to solve a issue in binaryornot cookiecutter/cookiecutter-django#1118 (comment)

Is expected that the chardet.detect() function return a dictionary with encoding and confidence keys. However, None is being returned for some cases.

The cases in question is if I try to detect the encoding of the first 1024 bytes of images files. (I do not know if there are other cases like that.)

The question is: Is this the expected result, is a issue or is a documentation issue?

From this snippet and this sample_images.zip:

import chardet

print('chardet version:', chardet.__version__)

def detect_from_file(filename, length=1024):
    first_chunk_of_file = None
    with open(filename, 'rb') as f:
        first_chunk_of_file = f.read(length)
        detected_encoding = chardet.detect(first_chunk_of_file)
        return detected_encoding


images = ['bmp_format.bmp', 'gif_format.gif', 'jpg_format.jpg', 'png_format.png', 'tga_format.tga']

for i in images:
    c = detect_from_file(i)
    print('____\nfile: {}'.format(i))
    print(c, '\n')

Chardet 3.0.1 return:

chardet version: 3.0.1
____
file: bmp_format.bmp
{'encoding': 'ISO-8859-1', 'confidence': 0.73, 'language': ''} 

____
file: gif_format.gif
None 

____
file: jpg_format.jpg
None 

____
file: png_format.png
None 

____
file: tga_format.tga
{'encoding': 'ISO-8859-1', 'confidence': 0.73, 'language': ''}

but the expected values to return is

chardet version: 2.3.0
____
file: bmp_format.bmp
{'encoding': 'windows-1252', 'confidence': 0.73} 

____
file: gif_format.gif
{'encoding': None, 'confidence': 0.0} 

____
file: jpg_format.jpg
{'encoding': 'ISO-8859-2', 'confidence': 0.2766991521036145} 

____
file: png_format.png
{'encoding': 'IBM866', 'confidence': 0.345640381912402} 

____
file: tga_format.tga
{'encoding': 'windows-1252', 'confidence': 0.73}

[dan-blanchard]

This was a completely accidental change on my part. I will push out a bugfix release ASAP.

[pydanny]

Thanks @dan-blanchard! 😄

In the meantime, we've pushed out a new version of binaryornot that forces the user to the older version of chardet. Not a perfect solution, but it will mitigate the problem temporarily.

[dan-blanchard]

@pydanny just released 3.0.2 with this fix.

[pydanny]

Hooray! Testing it now...

[pydanny]

Tests ran great. Just pushed up 0.4.3 of binaryornot.

Again, @dan-blanchard, thank you for the quick response. 😄 🍪

[luzfcb]

@dan-blanchard thank you for the hard work on this library 👍