Specify (in the model) the encoding

[iherman]

The current text in the model does not specify that the metadata and the content must be in UTF-8. It may be better to make this explicit.

(This came up via issue #551)

[JeniT]

UTF-8 is an encoding of Unicode characters in a byte stream. The model is an abstract model. There shouldn't be any necessity to talk about encoding in an abstract model.

There is obviously a necessity to talk about the encoding of Unicode characters within CSV files, but that's handled within 7.2 Encoding and using the encoding flag when parsing.

[danbri]

Q: what do we do about cases in which a widely used character set doesn't have perfect mapping of all its defined chars 1:1 into unicode chars?

apparently Shift_JIS has this issue. https://support.microsoft.com/en-us/kb/170559

Do we discourage Shift_JIS or just lossily convert? @JeniT in See http://www.w3.org/2015/05/27-csvw-irc#T14-40-39 argues to replace, following https://encoding.spec.whatwg.org/#concept-encoding-process https://encoding.spec.whatwg.org/#error-mode

[JeniT]

We discussed: and resolved to specify that CSV files are parsed based on their encoding, according to the encoding spec with replacement mode, but cell errors are added if the resulting cell string values contain a U+FFFD.

http://www.w3.org/2015/05/27-csvw-irc#T14-47-07

[6a6d74]

+1

[danbri]

+1

[iherman]

I would say +1, but have the admin issue of normative reference: the encoding spec of the WHATWG is not, as far as I know, reference-able normatively. Can we get around that? Isn't it enough to say that we expect UTF-8, although the http response may return other encodings?

[JeniT]

This won't be a normative reference, because the only things we ever say about parsing are non-normative. We already referencing the WHATWG encoding spec in a non-normative way.

[iherman]

On 27 May 2015, at 12:16 , Jeni Tennison notifications@github.com wrote:

This won't be a normative reference, because the only things we ever say about parsing are non-normative. We already referencing the WHATWG encoding spec in a non-normative way.

Pfew:-) +1 from then!

Ivan

—
Reply to this email directly or view it on GitHub.

---

Ivan Herman, W3C
Digital Publishing Activity Lead
Home: http://www.w3.org/People/Ivan/
mobile: +31-641044153
ORCID ID: http://orcid.org/0000-0003-0782-2704

[gkellogg]

@JeniT @iherman: just going through open branches, and saw that 27eaa52 on issue-557 branch was never merged. Did this get lost, or did we decide to do something different?

[JeniT]

I think it got lost. Can you get it merged in please? (As above, this is non-normative so purely an editorial change.)

[gkellogg]

Done.