Improve binary file check

[rasa]

Currently, we determine if a file is binary or not by sniffing the content type using the first 512 characters and seeing if it's text/ or application/octet-stream. This produces lots of false positives, such as files that are UTF16.

I think a better solution would be to determine the charset and then check for binary characters. I found that the tool dos2unix, uses the characters \x00-\x08,\x0b,\x0e-\x1f to determine if a file is binary. This is a very widely used tool, and this check seems reasonable. Thoughts?

[mstruebing]

Sounds good, I know that this part of this application is actually a weak point so improving it in any way would be great.

[rasa]

OK, I will see if I can craft a PR that addresses this. Kinda a pain point for us. Thanks for being open to this.

[mstruebing]

Thank you very much, a PR would be really great as I do not be able to spend so much time on this tool anymore as it sometimes deserves 👍