Open rnc files using UTF-8 encoding

[brechtm]

rnc2rng currently uses the platform's default encoding when reading rnc files. This causes platform-dependent issues: brechtm/citeproc-py#81. I think it would be safe to assume that rnc files will be UTF-8 encoded, so it would make sense to always load rnc files using UTF-8.

[djc]

Thanks for the feedback! Your proposal makes sense to me. I see you've already prepared a fix, do you want to submit that as a PR? I can merge it and publish a new release if that helps.

[brechtm]

I haven't been able to test the patch yet. I have asked @yurivict who ran into the issue to verify whether it works.

[brechtm]

From the RELAX NG Compact Syntax reference, it seems that rnc files loaded from the filesystem should indeed always be opened as UTF-8 of UTF-16, depending on the BOM:

The textual representation of the RELAX NG schema in compact syntax may be either a sequence of Unicode characters or a sequence of bytes. In the latter case, the first stage is to transform the sequence of bytes to the sequence of characters. The sequence of bytes may have associated metadata specifying the encoding. One example of such metadata is the charset parameter in a MIME media type [RFC 2046]. If there is such metadata, then the specified encoding is used. Otherwise, the first two bytes of the sequence are examined. If these are #xFF followed by #xFE or #xFE followed by #xFF, then an encoding of UTF-16 [Unicode] will be used, little-endian in the former case, big-endian in the latter case. Otherwise an encoding of UTF-8 [Unicode] is used. It is an error if the sequence of bytes is not a legal sequence in the selected encoding.

[brechtm]

Passing encoding='utf-8' when opening the file is not enough, since rnc files could also be UTF-16 encoded.

[djc]

I suspect we could use codecs.open(), instead.

[brechtm]

I'm not sure how that would work. It doesn't seem to be able to detect the encoding of the file. The RNC spec suggests that files are either UTF-8 of UTF-16 encoded. The latter only when a BOM is present. So we could open the file in binary mode and read the first two bytes to determine the encoding to use (UTF-8, UTF-16 BE or UTF-16 LE).

[brechtm]

codecs.open() handles newlines differently apparently, so I used io.open() instead.