core data types: string details, e.g. lone surrogate

[dckc]

In #5 (comment) @erights writes:

What does capnp do when an incoming string allegedly in UTF-8 encodes a lone surrogate?

Let's please have a test case for that.

In #5 (comment) @zenhack writes:

I'll make a proposal: ocapn strings are isomorphic to lists of integers in the range [0, 2^21).

[dckc]

@zenhack does this specification then justify a test where a JavaScript client sends "ABC" and a scheme client receives it and compares it to the string "ABC"?

Or is the protocol silent on whether the JavaScript client might want to rot13 everything?

Surely "ABC" here means "ABC" there... but then... the slope from "ABC" to unicode normal forms and CESU-8 is remarkably slippery... and impacts UI security in subtle ways.

[erights]

It is up to each endpoint definition to define the binding of the protocol data model to their internal language model, as long as they preserve round-tripping, i.e., the distinction between the equivalence classes. There should be nothing in the protocol definition, indeed cannot be, that prohibits a conforming endpoint from defining a language binding that transforms through ROT13, since it still round trips.

That said, this does create a separate obligation on those endpoint definitions to define that language binding. A language binding that specifies translation though ROT13 is, hopefully, a language binding no one uses.

[dckc]

ok, so a test for "ABC" in, "ABC" out is out of scope for the data model alone, but the test framework might include bindings on each end.

[erights]

Yes. Taking the language binding into account, in bounds is testing that "Something that maps to the protocol 'ABC' round trips to something that maps to the protocol 'ABC'", where "maps to" means "according to the language binding".

[erights]

My text is likely not great standards language. I'm happy with any rephrasing that means the same thing. (meta analogy intended ;) )

[erights]

@zenhack

I'll make a proposal: ocapn strings are isomorphic to lists of integers in the range [0, 2^21).

I really love @zenhack's proposal and hope we can keep it. But I found a problem with it we need to discuss. The proposal above would require that the distinction between a proper surrogate pair vs the unicode char it encodes be preserved. That this distinction round trip. Put another way, that the data model holds them to be in distinct equivalence classes.

For languages that rely on the UTF-8-like representation and encode/decode algorithm, that might be the least effort outcome. But for languages like Java and standard (non-XS) JS, their native string encoding is UTF-16. Their OCapN language binding can only map a proper surrogate pair to an OCapN encoding of those two 16 bit values, or it can map it to the one 21 bit value. It cannot do both. Whichever it does, the other kind of OCapN string does not round trip through any language string natural to that language.

We could loosen the round trip criteria to consider the two 16 values that happens to be a proper surrogate pair to be in the same equivalence class as the 21-bit value it UTF-16 encodes. IOW, if one of these representations round trips back as the other, that OCapN consider that a valid round trip.

One unpleasant thing about this answer: it would make it terribly painful to define a lexicographic order among such strings. The natural lexicographic order according to one representation would be different that the order according to another, allegedly equivalent, representation. IOW, the natural sort order would not be pass-invariant. (Note that comparison/sort order are not part of the core data model we're trying to agree on. It would just create pain for optional higher layers.)

Hopefully, someone in the room is more acquainted with Unicode than I, and can point out something I'm missing.

[dckc]

For reference, Agoric uses CESU-8. We have a test case resulting from a long discussion in an issue:

• test(xsnap): XS round-trips surrogate pairs Agoric/agoric-sdk#4920
• CESU-8 support in XS to address UTF-8/UTF-16 confusion risk Agoric/agoric-sdk#2118

Their OCapN language binding can only map a proper surrogate pair to an OCapN encoding of those two 16 bit values, or it can map it to the one 21 bit value. It cannot do both. Whichever it does, the other kind of OCapN string does not round trip through any language string natural to that language.

CESU-8 maps it to two 16 bit values "and then each surrogate code point is encoded in UTF-8."

It's somewhat like utf-8, but sufficiently different that you can't just wave your hands at the difference. guile scheme most likely has utf-8 support out of the box but not CESU-8. The HTML spec goes so far has to prohibit support for CESU-8 in 13.2.3.3 Character encodings

I wondered what rust folks do... there's a crate for that: cesu8 - Rust. But it says stuff like "Yes, this is ugly. Use of this encoding is discouraged by the Unicode Consortium. It's OK for working with existing internal APIs, but it should not be used for transmitting or storing data."

[dckc]

Idea: 2^21 is 0x200_000; the highest unicode code point is 0x10F_FFF. There are 0x800 surrogates. So a JS language binding could reserve, say, 0x180_000 to 0x180_7FF for surrogates.

How we encode them for transfer is a bit of a puzzle, but CESU-8 is one solution, as well as the more compatible WTF-8. Wikipedia notes PEP 383 as another solution.

I wonder how much utility we'll get out of this "abstract sequence over [0, 2^21)" notion, though. In tests, writing [65, 66, 67] instead of "ABC" will get old real quick. The test framework will most likely include a text encoding, and it'll be pretty hard to see that the encoding isn't quite part of the OCapN spec. To resolve this issue to my satisfaction, we ought to have a test that shows how we handle lone surrogates between Agoric, Spritely, and capn-proto (even if the answer is: "we don't").

[gibson042]

One unpleasant thing about this answer: it would make it terribly painful to define a lexicographic order among such strings. The natural lexicographic order according to one representation would be different that the order according to another, allegedly equivalent, representation. IOW, the natural sort order would not be pass-invariant. (Note that comparison/sort order are not part of the core data model we're trying to agree on. It would just create pain for optional higher layers.)

I am also concerned about this, and believe the correct ordering to be by Unicode code point rather than by UTF-16 code unit (IOW, native JavaScript string comparison gets it wrong; "\u{0FF41}" < "\u{1D306}" unfortunately evaluates to false because the U+1D306 TETRAGRAM FOR CENTRE code point is represented as the surrogate pair "\uD834\uDF06"). AFAICT, this is a family of latent bugs in Endo.

As for transmitting lone surrogates, both CESU-8 and WTF-8 declare theirselves to be inappropriate for interchange. It would be great to restrict strings to Unicode scalar values (which do not include surrogate code points U+D800 through U+DFFF) and use UTF-8 in the wire format, which is not too difficult to validate even in JavaScript that parses JSON:

// A sequence of text in which every component is
// non-surrogate non-backslash OR
// backslash followed by non-`u` OR
// `\uHHHH` for a non-surrogate OR
// a [high, low] surrogate pair in which either or both surrogates may be escaped.
const rAllSurrogatesPaired =
  /^(?:[^\uD800-\uDFFF\\]|\\[^u]|\\u(?:[^d]|d[0-7])|(?:[\uD800-\uDBFF]|\\ud[89ab][0-9a-f]{2})(?:[\uDC00-\uDFFF]|\\ud[c-f][0-9a-f]{2}))*$/i;

const parseUnicodeSafeJSON = jsonText => {
  rAllSurrogatesPaired.test(jsonText) || Fail`input contains unpaired surrogates`;
  return JSON.parse(jsonText);
};

[erights]

Some on this string discussion happened on the original thread starting at #5 (comment) . Please consider this a virtual include of that subthread in this thread.

[dpwiz]

What is the use case for sending things that are 1) unrepresentable in UTF8 2) have "text" semantics?

[dckc]

What is the use case for sending things that are 1) unrepresentable in UTF8 2) have "text" semantics?

JavaScript strings include unpaired surrogates...

"it should be able to encode any string representable in JSON, which therefore must cover such horrors as unpaired surrogates."

Whether these monsters round-trip thru ocapn is the subject of this issue.

[dpwiz]

JavaScript strings include unpaired surrogates...

Okaaay.... Why would one want to send such things as a textual data?

[dpwiz]

Well, of course, anyone can send whatever they want on the wire. So the recipients should (must, really) check incoming bytes and welcome only valid data. If, for some valid reason, someone wants to breach this contract, they should state that clearly: "Here's my data. Usually, it is UTF8-formatted, but can contain invalid sequences. Please preserve them for round-tripping." IMO this is a special case, not the "happy path" here.

UTF8 has SIMD'd validation routines than can guard against such things at negligible cost. And I expect every programming language worth their bytes to be able to do this or to have a library to deal with this. Infecting them with having to deal with some other platform sins is punishing the wrong side.