Support arbitrary encodings for XML documents

[Pytheas01]

I get the following error messages when I read a DOS-UTF8 XML file in a Rust program that uses rust-xml, on Windows 7:

Running `target\rust-xml-w`
Error: 1:2: Unexpected characters outside the root element: 

If I convert the XML file to ANSI LATIN 1, the issue disappears.

My version of rustc is

rustc 0.12.0-pre-nightly (0bdac78da 2014-09-01 21:31:00 +0000)

[netvl]

This, as you correctly identified in the comment to another issue, is likely a BOM mark which terminates the processing (BTW, there is no such encoding as DOS-UTF8, there is only just UTF-8).

Unfortunately, I don't think I want to add special handling for it in rust-xml. Ideally this should be handled on the low level by I/O streams library, and rust-xml should use properly decoded streams of characters. There is an encoding library, rust-encoding, which probably can be a basis for this, but I'm not sure if it provides proper streams and not only buffered decoding.

Another reason I don't want to add BOM handling directly into the library is that BOM mark should not be used in UTF-8 encoded files - it is not even recommended by W3C.

For now, your best bet will be dropping first character from the buffer if it is a BOM mark. However, as far as I'm aware, Rust Buffer does not provide an ability to "unread" characters (though it would be quite natural), so you will have to either to read your document into memory entirely and just slice it from the second byte onward, or you can write your own Buffer or even Reader implementation which filters out the first several bytes if they are UTF-8 BOM mark.

[Pytheas01]

Hi,

have you mentioned this to the Rust developers?

Philippe

----- Mail original -----
De: "Vladimir Matveev" notifications@github.com
À: "netvl/rust-xml" rust-xml@noreply.github.com
Cc: "Phiroc" phiroc@free.fr
Envoyé: Mardi 2 Septembre 2014 10:25:30
Objet: Re: [rust-xml] Issue with UTF8 files (#10)

This, as you correctly identified in the comment to another issue, is likely a BOM mark which terminates the processing (BTW, there is no such encoding as DOS-UTF8, there is only just UTF-8).

Unfortunately, I don't think I want to add special handling for it in rust-xml . Ideally this should be handled on the low level by I/O streams library, and rust-xml should use properly decoded streams of characters. There is an encoding library, rust-encode , which probably can be a basis for this, but I'm not sure if it provides proper streams and not only buffered decoding.

Another reason I don't want to add BOM handling directly into the library is that BOM mark should not be used in UTF-8 encoded files - it is not even recommended by W3C.

For now, your best bet will be dropping first character from the buffer if it is a BOM mark. However, as far as I'm aware, Rust Buffer does not provide an ability to "unread" characters (though it would be quite natural), so you will have to either to read your document into memory entirely and just slice it from the second byte onward, or you can write your own Buffer or even Reader implementation which filters out the first several bytes if they are UTF-8 BOM mark.

—
Reply to this email directly or view it on GitHub .

[netvl]

@PHIROC, there was an RFC on something like that, but it was postponed. I now looked more closely into rust-encoding and it seems to me that it has facilities which would allow implementing streaming decoding, but I'm not sure if I'm able to implement such thing, at least not right now.

[netvl]

There is a new library, encoding_rs, which explicitly supports streaming encoding/decoding. I guess it is possible to use it for streaming encoding/decoding.

[kornelski]

latin1, ASCII, and UTF-16 are now supported.

I'm not sure if there's demand for other encodings like legacy codepages or non-Unicode CJK encodings. Please open new issue if you need other encodings.