Issue parsing surrogate pair encoded emoji in pipeline JSON file

[jim]

We're seeing an issue where the buildkite agent fails to parse the JSON-formatted pipeline file we're passing to buildkite-agent pipeline upload:

Pipeline parsing of "tmpjdd2BG" failed (Failed to parse tmpjdd2BG: line 531: found invalid Unicode character escape code)

I was able to reproduce the issue on the Go Playground using the same YAML library that the agent uses. The Go standard library JSON parser handles the input without an issue.

I found a related issue on the upstream go-yaml project.

[triarius]

Hi, we can't commit to fixing at the moment, but we will accept back-ported fixes from upstream or elsewhere. In the mean time you could try to replace unicode emojis with buildkite's emoji syntax from https://github.com/buildkite/emojis. eg :bug: will be rendered as some form of 🐛.

[DrJosh9000]

I was having a play with this to see if the upgrade to gopkg.in/yaml.v3 (#1930) helped in any way. It does not.

But here's another maybe-workaround for you: yaml.v3 can parse UTF-16 strings, little- or big-endian, where such surrogate pairs are needed for the higher codepoints. (Hmm. Why use surrogate pairs in UTF-8?) There are even tests: https://github.com/go-yaml/yaml/blob/f6f7691b1fdeb513f56608cd2c32c51f8194bf51/decode_test.go#L742

But, the parser seems to rely on there being a byte-order mark (https://go.dev/play/p/EUgD5iJlI2m) even though it is supposed to be able match the correct encoding without one (https://yaml.org/spec/1.2.2/ section 5.2).

[123sarahj123]

A PR has been created to add support for surrogate pairs: go-yaml/yaml#1029