fix: input file encoding

[JCHacking]

input files in lock-format are expected in a certain encoding,
other input file encodings are detected.

fixes #448

[jkowalleck]

why did you use a niche library https://pypi.org/project/faust-cchardet
why not use a library that is widely used? like https://pypi.org/project/chardet ?

if it was for the py3.6 compatibility, chardet provides older compatible versions.

[JCHacking]

why did you use a niche library https://pypi.org/project/faust-cchardet
why not use a library that is widely used? like https://pypi.org/project/chardet ?

if it was for the py3.6 compatibility, chardet provides older compatible versions.

That was one of the reasons, the other one is because some encodings like cp1252 are returned as Windows-1252 so they are not exactly the same to pass it to the decode method.

Also faust-cchardet is a maintained fork of cchardet which is written in C so it has better performance.
Traducido con DeepL https://www.deepl.com/app/?utm_source=android&utm_medium=app&utm_campaign=share-translation

[jkowalleck]

re #596 (comment)

[...] it has better performance

Performance should not be a concern at this point.
Maintainability and support of used libraries is more important here. So better use a well-maintained library like chardet that is widely used.
If this choice is insufficient at any point, we can still report bugs to the maintainers team of that library.

---

see #448 (comment)

could you test the following on your system?

• for poetry.lock and Pipfile.lock use: open(..., encoding="utf8") -- no mode, must path and encoding
• for requirements.txt the char-detection is okay in general.

[JCHacking]

re #596 (comentario)

[...] tiene mejor rendimiento

El rendimiento no debería ser una preocupación en este momento. La mantenibilidad y el soporte de las bibliotecas usadas son más importantes aquí. Así que es mejor utilizar una biblioteca bien mantenida como chardetla que se utiliza ampliamente. Si esta elección es insuficiente en algún momento, aún podemos informar errores al equipo de mantenimiento de esa biblioteca.

ver #448 (comentario)

¿Podrías probar lo siguiente en tu sistema?

• para poetry.locky Pipfile.lockuso: open(..., encoding="utf8")-- sin modo, debe ruta y codificación
• para requirements.txtla detección de caracteres está bien en general.

Perfect, then I change the library to chardet with a version that supports python 3.6 and then I try the other option that you mention of only validating the coding in requirements.txt

[JCHacking]

The change is already done

Basically, if the open is done in byte mode the encoding will be inspected, otherwise it will be assumed that everything is OK.

And regarding chardet I have made a replace to make it work with windows, since it returns Windows-1252 but python only understands cp1252.

[jkowalleck]

I had to do minor version range adjustments and other chores.

I will add a regression test, and then, this fix is ready to go.
Thank you for your effort, @JCHacking

[JCHacking]

I had to do minor version range adjustments and other chores.

I will add a regression test, and then, this fix is ready to go.
Thank you for your effort, @JCHacking

Thanks to you for letting me collaborate in its solution.

[jkowalleck]

fix is available as of v3.11.3