Make the Source class report the encoding #677
This pull request is the first of a series to provide multibyte encoding support to readr on the tokenizers. I made a first attempt in #673, but that PR was too big to be easily reviewed.
With this pull request, the datasource() R function and the associated C++ Source class now will require an encoding. If this encoding is ambiguous in endianness (like UTF-16 or UTF-32, which mandate a Byte Order Mark) the BOM is detected and skipped (as happened before) and the encoding is updated to reflect the endianness (UTF-16LE, UTF16-BE...).
Future pull requests will use this encoding for proper source parsing.
As this PR is not a new feature (not yet at least) and it is not a bug fix, I have not added a NEWS entry. Feel free to ask for it if you feel it is better to have it.
Once this is merged I will submit another PR, that offers the possibility of multiple comments and doing the comment parsing in the Source class (instead of duplicated code in Source and Tokenizers):
First, I wanted to thank you for working on this PR and the previous ones. This was not a simple problem and I appreciate the effort you went through to solve it.
I also wanted to apologize for leaving this open for so long, it was certainly not my intent to keep you waiting, but other projects and priorities kept me busy.
We have plans to begin moving away from the legacy parser in readr, in favor of the parsers in vroom or possibly arrow. Because of this I don't think it would be prudent to add this amount of new code to the existing parser.
Developing good software requires relentless focus, which means that we have to say no to many good ideas.
Even though we're closing this issue, we really appreciate your work, and hope you'll continue to contribute in the future.