encoding/json/jsontext: support 32-bit floating-point numbers

[ChrisHines]

Go version

go1.25.x

Output of go env in your module/workspace:

GOEXPERIMENT=jsonv2

What did you do?

func main() {
	var (
		pi32 = float32(math.Pi)
		pi64 = float64(math.Pi)
	)

	m := map[string]any{
		"f32": pi32,
		"f64": pi64,
	}

	out, _ := json.Marshal(m, jsontext.WithIndent("    "))
	fmt.Printf("%s\n", out)

	enc := jsontext.NewEncoder(os.Stdout, jsontext.WithIndent("    "))
	enc.WriteToken(jsontext.BeginObject)

	enc.WriteToken(jsontext.String("f32"))
	enc.WriteToken(jsontext.Float(float64(pi32)))

	enc.WriteToken(jsontext.String("f64"))
	enc.WriteToken(jsontext.Float(pi64))

	enc.WriteToken(jsontext.EndObject)
}

What did you see happen?

The above program prints:

{
    "f32": 3.1415927,
    "f64": 3.141592653589793
}
{
    "f32": 3.1415927410125732,
    "f64": 3.141592653589793
}

Notice how json.Marshal nicely formats the float32 version of pi with a correct number of significant digits while the most obvious way to produce the same output with jsontext.Encoder formats it with a misleading (and wrong) set of digits.

I discovered this while experimenting with retrofitting the json implementation of a structured logging encoder to use jsontext. Although the logging encoder in question is not public its encoding interface is quite similar to the zapcore.ObjectEncoder which has an AddFloat32(key string, value float32) method.

Our log encoder does not use the json package but does have unit tests that ensure it produces the same output as an equivalent json.Marshal call for all the primitive types we support. Using jsontext.Encoder it was trivial to match the output for all types except float32.

Note that in order to match the json.Marshal behavior for floating point values our current log encoder has a copy of an older version of the code that json/v2 has in the encoding/json/internal/jsonwire.AppendFloat function.

What did you expect to see?

I think the jsontext package needs better support for float32 tokens.

For use cases like the above there should be a way to construct a jsontext.Token that carries a float32 value in a way that the Encoder eventually calls jsonwire.AppendFloat with 32 for its bits parameter.

[ChrisHines]

cc: @dsnet, @aclements

[gabyhelp]

Related Issues

• encoding/json: float32 may not convert correctly #34559 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[dsnet]

RFC 8259 section 6 suggests that JSON numbers are widely considered to be IEEE 754 double-precision numbers. Of note, the text-based format of JSON provides no way to reliably know when parsing whether the number was originally a float16, float32, float64, or even an arbitrary precision floating-point number.

I'm a bit uncomfortable with the idea of adding a func Float32() Token constructor because that calls into question whether there should be a func (Token) Float32() float32 accessor as well. However, the latter would be impossible to reliably distinguish for decoded tokens whether someone should be using the float32 or float64 accessors.

I'm more open to the idea of an AppendFloat or AppendNumber helper function since it's more clearly unidirectional. You would have to go through the raw Value representation instead of the opaque Token representation, but I thinks okay for this edge case.

Of note, there is a well defined standard for how to canonically encode float64 in JSON, which is defined in section 7.1.12.1 of ECMA-262. However, there is no well-defined standard for how to canonically encode float32 in JSON. The current formatting of float32 was added in CL/30371 and more or less uses the same algorithm as float64, but chooses cutoff values that are more specific to float32. I suspect that if there were ever a canonical representation of float32 chosen, it would be identical to what we use.

[ChrisHines]

I'm more open to the idea of an AppendFloat or AppendNumber helper function since it's more clearly unidirectional. You would have to go through the raw Value representation instead of the opaque Token representation, but I thinks okay for this edge case.

I worry that solution makes the correct way to encode a float32 value harder and less discoverable than the wrong way. Can you provide a concrete example of what the API and an example use of it might look like?

[dsnet]

We discussed this at length at the json/v2 working group meeting...

To summarize the issue, consider how the following are encoded:

pi32 := float32(math.Pi)
pi64 := float64(math.Pi)

enc := jsontext.NewEncoder(os.Stdout)

json.MarshalEncode(enc, pi32) // 3.1415927         (minimal, but accurate representation of Pi as a float32)
json.MarshalEncode(enc, pi64) // 3.141592653589793 (minimal, but accurate representation of Pi as a float64)

enc.WriteToken(jsontext.Float(float64(pi32))) // 3.1415927410125732 (excessively long, but accurate representation of Pi as a float32)
enc.WriteToken(jsontext.Float(float64(pi64))) // 3.141592653589793  (minimal, but accurate representation of Pi as a float64)

Several notes:

• At the core, there is a fundamental mismatch between how computers represent floating-point numbers and how JSON represents them. Computers use base-2, while humans use base-10. The conversion from base-2 to base-10 leads to a perceived loss of precision for some values due to truncation and rounding.

  • For example, encoding float64(float32(math.Pi)) results in 3.1415927410125732, where the trailing 10 digits are different from the real value of Pi in base-10. This is a perceived inaccuracy to humans since we don't know that this only contains 32 bits of precision (nor can we compute that in our head). Unfortunately, the expected precision is not encoded in the format.

  • There is no loss of precision if you do know that this is precise only to 32 bits. So for example, if you do float32(must.Get(strconv.ParseFloat("3.1415927410125732", 32))) == float32(math.Pi), it will report equality.

  • The ideal encoding of a float depends on an implicit agreement of the expected bit-width between the sender and receiver. (It is technically possible to encode floats as base-10 without the receiver needing to know the original bit precision, but this is almost never done; see NOTE1).

• The various JSON standards do not specify that JSON numbers are float64, but heavily imply that float64 is the primary representation of JSON numbers (given JSON's heritage in JavaScript). For this reason, jsontext.Float today uses 64-bit precision and why json.Unmarshal into an any type chooses a float64 for JSON numbers.

• JSON defines an algorithm for how to exactly encode a 64-bit float (see RFC 8785). Most JSON libraries output this exact format. Of note, there is no standard for how to canonically encode a 32-bit float in JSON.

• The "json" package uses a shorter representation for a float32, but the "jsontext" package does not (it always encodes assuming the original bit-width is 64). This issue aims to resolve this discrepancy. For the 32-bit encoding, the "json" package uses a modification of the RFC 8785 algorithm, but with thresholds adjusted for 32 bits. Of note, this means that Go's representation of 32-bit floats is non-standard.

• There is no natural way to exactly represent a JSON number in Go (except to use math/big.Float, which is a large dependency). The existing Int, Uint, and Float constructors/accessors in jsontext is a recognition of this fact, where we try to represent a JSON number as the most common numeric types in Go. This issue argues for functionally decomposing Float (e.g., by adding a Float32 constructor) to better control bit precision.

• If we add a Float32 constructor, do we need a Float32 accessor? On one hand, it seems that you could always do float32(Token.Float()). However, this can lead to rounding errors of the least-significant bit (LSB). It is a very minor error, but it means that values are not round-trip exact (see NOTE2 for more details). Thus, we want to add a Float32 accessor as well.

Thus, we are leaning towards adding the following:

func Float32(float32) Token
func (Token) Float32() float32

These are a constructor/accessor pair similar to Float, but operate on 32 bits instead.

We could consider renaming the existing Float constructor/accessor as Float64:

• An argument for would be to have consistency with Float32.
• An argument against is that it would be inconsistent with Int and Uint, which don't have the 64 suffix.
• An argument for would be that JSON generally recommends JSON numbers being float64 and the shorter name makes the 64-bit variant more prominent.
• An argument against would be for clarity when reading. If bit-width matters, then always seeing 32 or 64 makes that clear.

We could also consider adding the following:

func AppendFloat(b []byte, f float64, bits int) []byte

The formatting of floating-point numbers in JSON is not trivial. As mentioned earlier, the encoding of 64-bit floats follows a standard, while encoding of 32-bit floats would be a Go-specific representation (though a reasonable extension of RFC 8785).

[NOTE1] Technically, you can perfectly convert a base-2 floating-point number to base-10 without any loss of precision since any fractional base-2 value (i.e., 1/2ⁿ) is a terminated decimal that can be exactly represented in base-10 with a finite number of digits. A floating-point number is essentially a summation of fractional base-2 values. Encoding in base-10 exactly would end up with a huge (but finite) number of digits. Most floating-point conversion libraries do not do this. They instead only emit enough base-10 digits such that parsing it back into a float preserves the original value assuming that you know the original bit-width.

• For example, here's a demo that computes the exact value of Pi when stored in a float32 or float64. Of note, the trailing digits look different from the exact value of Pi in base-10 because this is exactly representing the imprecision that occurs with float32 or float64.

[NOTE2] One might expect float32(strconv.ParseFloat(s, 32)) == float32(strconv.ParseFloat(s, 64)) to always be true, but it isn't:

• Here are some counter examples.
• The reason why this property doesn't hold is because fractional base-10 cannot be exactly represented in base-2 using a finite number of bits. As a simple example, the number 0.1 in base-10 is 0.0001100110011... (with 0011 repeating) in base-2. This means that converting base-10 to base-2 requires rounding (and implies loss of precision). When converting float64 to float32, there is another rounding operation. This is double rounding. If the first conversion rounded up, that may cause the second rounding to round up, when it should have actually rounded down (and vice-versa).
• Converting a float32 to float64, encoding it according to RFC 8785 (the way almost all JSON libraries format a float64), parsing it as float64, and then casting to float32 never runs into conversion problems. Thus, doing WriteToken(Float(float64(f32))) on the sender and float32(ReadToken().Float()) on the receiver would always be accurate.
• Encoding a float32 according how the "json" package does it (which is a modification of the RFC 8785 algorithm to work with 32 bits), parsing it as float64, and then casting to float32 never runs into conversion problems except for a single number (i.e., 7.038531e-26) in the entire number space (wow, seriously?). Thus, doing WriteToken(Float32(f32)) on the sender and float32(ReadToken().Float()) on the receiver could be inaccurate. Instead, the receiver should be doing ReadToken().Float32().