2217 bin:decode-string: Input encoding

[ChristianGruen]

Before discussing this PR, we should have a look at #2221.

Closes #2217

[michaelhkay]

Typo utf16be -> utf-16be

[michaelhkay]

As noted during the meeting this week, there are problems deciding how to handle a BOM that is not at the start of the input (given that the function allows decoding starting at an offset other than zero).

It was also noted (if I heard correctly) that it might be useful to extract the encoding detection logic into a separate function.

[ChristianGruen]

We should also look at offsets that…

• do not point to the first byte of a multi-byte character, or
• point to a byte inside a BOM.

[ndw]

It feels like we're rubbing up against some unresolvable problems. Perhaps we should push some of this effort back on the user. After all, an offset of [1] when the string begins with a BOM is either exactly what the user wants to do or it's very, very likely an error, and I don't think we can tell which.

I think it was Juri who proposed an inspect function, let's say bin:inspect-binary. Off the top of my head:

bin:inspect-binary(
  $in as (xs:hexBinary | xs:base64Binary)?,
  $options as record(encoding as xs:string?, identify-characters as xs:boolean?, *)
) as record(has-bom as xs:boolean?, encoding as xs:string?, character-indexes as xs:integer*, *)

If you pass in an encoding in the options, then that's the encoding that's used and returned. If you don't, then the implementation uses heuristics to infer an encoding, which it returns. If the encoding is an appropriate UTF encoding, then the presence (or absence) of the BOM is returned in has-bom.

If identify-characters is explicitly true(), then character-indexes is a sequence (or array?) of offsets to the positions where characters start. (I think that's only useful for UTF encodings, but I could be wrong.)

It's a fairly expensive function to call, perhaps, but it means you, the user, can work out if you need to skip the BOM, or align a substring at a position where a character starts.

The bin:decode-string function can take an explicit encoding and it it uses that encoding and if the sequence of characters selected isn't usable in that encoding, an error is raised.

Is that at all sensible?

[ChristianGruen]

Is that at all sensible?

Possibly ;) The main reason why we got here was that we were looking for a function that decodes a binary-with-an-optional-BOM to a string. Do you think that bin:inspect-binary could help us here?

[ndw]

The hard way would be, check if there's a BOM, if there is, decode starting at offset [3].

But I think we could special case the "decode this whole UTF-x string and discard the BOM" behavior if the encoding is UTF-x, begins with a BOM, and no starting offset was given.

[ChristianGruen]

But I think we could special case the "decode this whole UTF-x string and discard the BOM" behavior if the encoding is UTF-x, begins with a BOM, and no starting offset was given.

Yes, I agree. We can either raise an error if an offset without encoding is supplied, or assume UTF-8.

[michaelhkay]

I'm inclined to say let's leave bin:encode-string and bin:decode-string as we left them after the previous discussion on this in issue #1751, and add a new function bin:infer-encoding(xs:hexBinary | xs:base64Binary) as xs:string which returns an encoding as a string, following defined rules.

If the user wants to drop a BOM appearing at the start of the string (or anywhere else), they can do it by hand.

[Arithmeticus]

bin:detect-encoding would be a better name, IMO.

[ndw]

I propose the signature

fn:inferred-encoding($data as (xs:hexBinary|xs:base64Binary),
                     $default-encoding as xs:string? := ()) as xs:string?

I think the user should be able to specify a default to use if inference fails and I think the function should be allowed to return () if it can't infer an encoding.

I prefer inferred-encoding over detect-encoding because I think it's clearer that the detection may be based on heuristics and might be incorrect. (Unless I've misunderstood the proposed semantics.)

[Arithmeticus]

I think for most speakers of English (native or nonnative) the two are synonyms. "Infer" was more popular of the two words in the 19th c., but it has been eclipsed by "detect" (see NGram visualization). A case in point: @ndw preferred the word "detection" over "inference" to describe what the function should do.

[ndw]

My resident linguist agrees with me, but I think technically this is a good way forward and I'm not going to lie down in the road over the name. Believing that you know the semantics of a function based entirely on its name is going to lead you astray sometimes.

[ChristianGruen]

I'm inclined to say let's leave bin:encode-string and bin:decode-string as we left them after the previous discussion on this in issue #1751

An additional function to infer the encoding looks reasonable to me.

In addition to that, I think we still need a straightforward solution. Imagine that one would always need to write things like…

let $encoding := bin:infer-encoding($bin)
let $offset := if($encoding = 'utf-8') then 3 else
  if(starts-with($encoding, 'utf-16')) then 2 else 0
return decode-string($bin, $encoding, $offset)

This applies even more if we want to recommend bin:decode-string as a shortcut for converting arbitrary binaries to strings. Many users are not even aware that a string-supplying blob can have something like initial bytes that define the encoding of the remaining input. – Let’s see what I can offer.

[ndw]

I think we could say that if the decode-string function is being asked to decode the whole string (no offset or encoding provided), and if the inferred (or detected!) encoding is UTF-x and it has a BOM, the BOM is removed from the resulting string.

It's going to be convenient for many users and it's easy to work around if you want the BOM or if you want to do something special simply by providing an offset of 0 or an encoding.

[ChristianGruen]

I prefer inferred-encoding over detect-encoding …

A general question may be what exactly we expect the function to do: Should it focus on the BOM patterns, or should it be allowed to take advantage of implementation-dependent rules or heuristics (in alignment with fn:unparsed-text)? In the latter case, an implementation might decide to always return an encoding, even if it is wrong.

[ndw]

I imagined that the semantics of (inferred|detect)-encoding is that it does anything it wants, right down to rolling dice, to work out an encoding. It returns what it decides, or () if it decides not to decide.

[michaelhkay]

unparsed-text() has extra information to work with, for example HTTP headers (and I still lament an operating system I used to work with that allowed you to record the encoding in the file descriptor...). So allowing implementation-dependent heuristics for unparsed-text() makes sense. decode-string only has a sequence of octets to work with, so there's much less reason to let different implementations do things differently.

[ChristianGruen]

I imagined that the semantics of (inferred|detect)-encoding is that it does anything it wants, right down to rolling dice, to work out an encoding. It returns what it decides, or () if it decides not to decide.

In that case it could make sense to return a sequence of encoding values, along with confidence values. The ICU CharsetDetector.detectAll method provides a very similar functionality.

[ChristianGruen]

It may be fair to question how relevant heuristics are today. At least on websites, UTF-8 is the clear winner (and Shift-JIS, which can be detected fairly well, is hardly used anymore): https://w3techs.com/technologies/overview/character_encoding

[ChristianGruen]

• See Function to detect/infer the string encoding from a binary #2250 for a separate function to detect/infer the encoding.
• See 2221 fn:unparsed-text: Encoding, BOM handling #2249 for an intent to clarify the BOM/encoding handling for fn:unparsed-text.

[ChristianGruen]

Revised; aligned with the updated fn:unparsed-text rules.

[michaelhkay]

"If a Unicode encoding is determined". What is a "Unicode encoding"? An encoding is a way of representing Unicode codepoints as octets - aren't all encodings "Unicode encodings"?

[I see that fn:unparsed-text also uses this term].

[michaelhkay]

It would be useful to have examples where a non-zero offset is supplied, with and without an encoding, with and without a BOM.

[ChristianGruen]

"If a Unicode encoding is determined". What is a "Unicode encoding"? An encoding is a way of representing Unicode codepoints as octets - aren't all encodings "Unicode encodings"?
[I see that fn:unparsed-text also uses this term].

Thanks, I’ll change it to UTF encoding. Note that “Unicode encoding” is also used in the Serialization spec (→ #2195 (comment)).

[ndw]

At meeting 140, the CG agreed to merge this PR.