UTF-8 BOM produces 'parse error: Invalid numeric literal'

[jklaiho]

I attempted to pipe some JSON from cURL to jq but got the aforementioned parse error. After validating the JSON with JSONLint and some initial confusion I noticed that the JSON data had the UTF-8 representation of the BOM character (U+FEFF), the three bytes 0xEF 0xBB 0xBF, at the beginning. If I stripped them out before piping to jq, everything worked.

I haven't tried if the same happens with, say, UTF-16 (little or big endian) encoded data, but either way, jq needs to handle the BOM correctly.

[stedolan]

jq doesn't handle UTF-16 documents. The BOM makes no sense at all in UTF-8 documents, how was that document generated?

On the other hand, the BOM can't be misinterpreted as anything else, so it wouldn't break anything to silently ignore one at the start of the document.

[jklaiho]

From Wikipedia: "The Unicode Standard permits the BOM in UTF-8, but does not require or recommend for or against its use. Byte order has no meaning in UTF-8, so its only use in UTF-8 is to signal at the start that the text stream is encoded in UTF-8."

The JSON feed is generated by a third party and I have no control over it. But since a BOM is valid (if pointless) in UTF-8, jq should handle it, and I suppose the silent ignoring approach is just fine.