Precision loss on vim.json.encode with integers having more than 14 digits

[mfussenegger]

Problem

:lua =vim.json.decode(vim.json.encode(3053700806959403))

Returns 3.0537008069594e+15, which is 3053700806959400

JSON floating number encoding is in general a bit messy but I think at least the default precision should be raised.
It's currently set to 14 in

neovim/src/cjson/lua_cjson.c

Line 86 in 1ee905a

#define DEFAULT_ENCODE_NUMBER_PRECISION 14

Other implementations try 15, and if that changes the result, they go up to 17, see:

https://github.com/DaveGamble/cJSON/blob/cb8693b058ba302f4829ec6d03f609ac6f848546/cJSON.c#L572C20-L572C20

, e.g. from wikipedia:

Numbers in JSON are agnostic with regard to their representation within programming languages. While this allows for numbers of arbitrary precision to be serialized, it may lead to portability issues. For example, since no differentiation is made between integer and floating-point values, some implementations may treat 42, 42.0, and 4.2E+1 as the same number, while others may not. The JSON standard makes no requirements regarding implementation details such as overflow, underflow, loss of precision, rounding, or signed zeros, but it does recommend expecting no more than IEEE 754 binary64 precision for "good interoperability"

Looking at the IEEE_754 article:

The 53-bit significand precision gives from 15 to 17 significant decimal digits precision (2−53 ≈ 1.11 × 10−16). If a decimal string with at most 15 significant digits is converted to the IEEE 754 double-precision format, giving a normal number, and then converted back to a decimal string with the same number of digits, the final result should match the original string. If an IEEE 754 double-precision number is converted to a decimal string with at least 17 significant digits, and then converted back to double-precision representation, the final result must match the original number.[1]

This popped up via: mfussenegger/nvim-dap#1004 - the dart debug adapter currently uses 53bit integers

I'm not sure how we deal with this given that cjson is vendored.
There is an upstream patch in that area, but it left the default: openresty/lua-cjson@f79aa68

Steps to reproduce

diff --git a/test/functional/lua/json_spec.lua b/test/functional/lua/json_spec.lua
index 25fdb48ee..7021f7f1e 100644
--- a/test/functional/lua/json_spec.lua
+++ b/test/functional/lua/json_spec.lua
@@ -57,10 +57,11 @@ describe('vim.json.decode()', function()
     eq(-100000, exec_lua([[return vim.json.decode('-100000')]]))
     eq(100000, exec_lua([[return vim.json.decode('  100000  ')]]))
     eq(-100000, exec_lua([[return vim.json.decode('  -100000  ')]]))
     eq(0, exec_lua([[return vim.json.decode('0')]]))
     eq(0, exec_lua([[return vim.json.decode('-0')]]))
+    eq(3053700806959403, exec_lua([[return vim.json.decode('3053700806959403')]]))
   end)

   it('parses floating-point numbers', function()
     -- This behavior differs from vim.fn.json_decode, which return '100000.0'
     eq('100000', exec_lua([[return tostring(vim.json.decode('100000.0'))]]))
@@ -145,10 +146,11 @@ describe('vim.json.encode()', function()
     eq('10', exec_lua([[return vim.json.encode(10)]]))
     eq('-10', exec_lua([[return vim.json.encode(-10)]]))
   end)

   it('dumps floats', function()
+    eq('3053700806959403', exec_lua([[return vim.json.encode(3053700806959403)]]))
     eq('10.5', exec_lua([[return vim.json.encode(10.5)]]))
     eq('-10.5', exec_lua([[return vim.json.encode(-10.5)]]))
     eq('-1e-05', exec_lua([[return vim.json.encode(-1e-5)]]))
   end)

and run tests

Expected behavior

Tests pass and vim.json.decode(vim.json.encode(3053700806959403)) doesn't loose precision

Neovim version (nvim -v)

NVIM v0.10.0-dev-752+g1ee905a63

Vim (not Nvim) behaves the same?

no, has no vim.json

Operating system/version

Linux

Terminal name/version

alacritty

$TERM environment variable

alacritty

Installation

From source

[justinmk]

I'm not sure how we deal with this given that cjson is vendored.
There is an upstream patch in that area, but it left the default: openresty/lua-cjson@f79aa68

We don't support json_cfg_encode_number_precision or any other runtime options since 8d4a53f .

Why wouldn't we just change the default?

[christopherfujino]

Why wouldn't we just change the default?

That would work too for this use case. @justinmk would you accept a PR for 17?