Handling a BOM with .json()

[pjirsa]

Anybody know how I should be handling a BOM with a returned json payload? Right now i'm getting this error when calling res.json()
Exception: FetchError: invalid json response body at https://api_that_returns_json_with_bom reason: Unexpected token ? in JSON at position 0

I tried adding an Accept header of application/json; charset=utf-8 but that didn't resolve the issue.

[jimmywarting]

Maybe something like this:

let text = await res.text()
if (text.charCodeAt(0) === 0xFEFF) {
  text = text.substr(1)
}
const json = JSON.parse(text)

Just a quick question: is native browser fetch able to handle it?

[bitinn]

I don't think BOM in JSON is valid, even if browsers might handles it. Going to label this as won't fix.

ref: https://stackoverflow.com/questions/4990095/json-specification-and-usage-of-bom-charset-encoding

[Jabbar-Zoop]

let text = await res.text()
try {
return JSON.parse(text)
} catch(error) {
console.error("Error:", err);
}

How about this solution @jimmywarting

[jimmywarting]

Hmm, chrome seems to be able to decode BOM

await new Response(String.fromCharCode(0xFEFF) + '1').json() // 1

What we are essentially doing is:

const buffer = Buffer.from(String.fromCharCode(0xFEFF) + '1'
JSON.parse(buffer.toString()) // Error

We should stop using buffer and turn towards using TextDecoder instead, this removes the BOM

JSON.parse(new TextDecoder().decode((Buffer.from(String.fromCharCode(0xFEFF) + '1')))) // 1

While we are at it:

async text() {
  let text = ''
  const decoder = new TextDecoder()
  for await (const uint8 of this.body) {
    text += decoder.decode(uint8, { stream: true })
  }
  text += decoder.decode() // flush
  return text
}

json() {
  return this.text().then(JSON.parse)
}

[rpaterson]

A more terse workaround:

const json = JSON.parse((await res.text()).trim());

I think calling trim() on a String of JSON should always be safe.

[jimmywarting]

Actually i think using textencoder is the right way to handle it.

[github-actions]

🎉 This issue has been resolved in version 3.2.3 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[ralyodio]

THis fixes it for me. was getting a BOM with authorize.net api and cloudflare workers:

const data = JSON.parse((await res.text()).trim());