Buffer.toString behavior change between v8 and v10

[jason-codaio]

• Version: v10.14.1
• Platform: MacOS Mojave 10.14.1 (18.2.0 Darwin Kernel Version 18.2.0)
• Subsystem: Buffer

I apologize if this is the wrong place, but I've spent part of my day debugging some checksum diffs when upgrading our system from Node v8.11.1 to Node v10.14.1 and noticed that there is a change in behavior for Buffer.toString() for invalid utf8. (Example below) Nothing really jumped out to me in https://nodejs.org/en/blog/release/v10.0.0/ notes. I've also noticed a floating point precision change, and Date format changes. Is there a better documentation for braking changes between versions?

On Node v10.14.1

> b = Buffer.from([0xED, 0xB8, 0x08])
<Buffer ed b8 08>
> b.toString()
'��\b'

On Node v8.11.1

> b = Buffer.from([0xED, 0xB8, 0x08])
<Buffer ed b8 08>
> b.toString()
'�\b'

[addaleax]

I think this change, and likely the other changes you’re mentioning as well, are a result of updating the V8 engine from 6.2 to 6.8; that is also likely the main reason why they aren’t explicitly mentioned in our (own) changelogs/release notes.

I guess it might make sense for us to include some details on V8 changes in our release notes, if that’s feasible? /cc @nodejs/v8

[ryzokuken]

@addaleax I think it's a great idea, since (1) it would help quell a lot of similar issues that are honestly too tricky to just guess unless you're dealing with this stuff on a near-daily basis, and (2) it could be integrated into our existing V8 upgrade toolchain, making the daily effort required to keep this going close to zero.

[BorntraegerMarc]

Just FYI that important bugs can come up from underlying V8 changes. For example I've been using the .toString() method to generate encryption keys. After updating node the keys changed suddenly and resulted in decryption errors. Same happened in this issue: #18075

I realize now that we should use a binary-to-text encoding for these kind of things. But still I would have saved a lot of hours if this V8 change would have been documented in the nodejs release notes.

[addaleax]

@BorntraegerMarc By now, we tend to include at least some information about changes of this kind in the release notes, which I hope is going to be helpful (although even we don’t always know about subtle breaking changes in V8 if they aren’t caught by our tests).

There’s nothing we can do about this particular issue anymore, so I’ll close this out.