readFile family of functions should avoid or document their use of system locale?

[Ptival]

relude/src/Relude/File.hs

Line 48 in 57c865d

readFileText = liftIO . T.readFile

uses Data.Text.IO's readFile.

This might be a source of problem, as described here:
https://www.snoyman.com/blog/2016/12/beware-of-readfile

Solutions could include

• switching to Data.ByteString.IO (as is done in http://hackage.haskell.org/package/classy-prelude-1.5.0/docs/src/ClassyPrelude.html#readFile possibly providing a readFileUtf8 convenience)
• being more explicit about this issue in the name of the function (readFileWithSystemLocale?)
• being more explicit about this issue in the documentation of the function.

Right now, someone would have to read the Relude.File documentation, click on the link to the Data.Text.IO documentation, and then scroll up to see the text about locale.

[chshersh]

@Ptival Documentation improvements are always appreciated. I don't think that switching to Data.ByteString.IO is an option because there is already readFileBS function if you prefer to work with ByteString. One of the relude goals is to be flexible and convenient. And sometimes it's more convenient to read file as a Text and work with Text.

It's good to know that reading file as a ByteString is a more correct and faster way of doing this operation. I think documentation of Relude.File can be enhanced.

[llelf]

https://www.snoyman.com/blog/2016/12/beware-of-readfile

Everyone links that, but it's very bad article.

• of course it depends of locale, see POSIX;
• of course using ByteStrings is faster, because there's no checking for validity of encoding;
• the worst of all is recommending lenientDecode, that quite possibly be a security issue later on.

[chshersh]

@llelf Do you know any alternative to this blog post? If no, I'm okay with mentioning this blog post in the documentation for this module because it contains useful information. I personally don't expect from people programming in Haskell to be aware of all low-level details how OS works.

relude doesn't have a goal to implement the safest and convenient framework for reading and writing files. It just re-exports existing functions and lifts them to MonadIO. Knowing which function to use in which situation is out of the scope for relude roadmap. However, mentioning already written blog post is not hard, documentation improvements are always appreciated.

[llelf]

I would just go with something like this (without any links):

These functions are for working with textual data, and are system and locale-sensitive (encoding, line-endings).
If you want binary data, use ByteString functions. You can then decode that data with the help of functions from Data.Text.Encoding module, e. g. decodeUtf8.

But it's your prelude :)