In-Game: json_parse(), json_decode(), string_lower(), string_upper() and other date-to-string functions can give errors when using a locale which uses commas for decimals

[gm-bug-reporter]

Description

json_parse() seems to have issues while parsing JSON files that have values written as a decimal number:

Script_Prepare() 
ERROR!!! :: ############################################################################################
ERROR in action number 1
of Create Event for object <undefined>:
JSON parse error : number expected
 at gml_GlobalScript_Script1 (line 21) - var _struct = json_parse(_json)
############################################################################################
gml_GlobalScript_Script1 (line 21)

This used to work fine in at least 2023.11.

Manually editing the file to change those decimals into integers or strings fixes the error.

Steps To Reproduce

-Run the project
-It will save a json file and try to load it immediately after
-The error appears upon trying to parse the loaded file

Which version of GameMaker are you reporting this issue for?

IDE v2024.600.0.560 Runtime v2024.600.0.577

Which operating system(s) are you seeing the problem on?

Windows 10.0.22631.0

Which platform(s) are you seeing the problem on?

Windows

3ddef4ce-96fc-44fd-a878-eff8ff6f6a87

[YYDan]

Your original project here and my slightly edited one so I can see the code actually works when there is no error both run absolutely fine for me in Beta IDE v2024.600.0.560 Beta Runtime v2024.600.0.577 and also in 2024.4.1 Monthly.

json_trouble-Dan.zip

[DiasFranciscoA]

I've been unable to replicate the issue on my system using Beta IDE v2024.600.0.560 and Beta Runtime v2024.600.0.577. However, I noticed a potential source of the problem—it seems you are using buffer_text instead of buffer_string. The buffer_text mode writes text to the buffer without appending the null terminator \0, which might cause issues when reading the data. If the buffer contains extra data beyond the text or if the subsequent memory isn't zeroed out, it could lead to the presence of extraneous data. This can render the text unreadable by json_parse. To avoid this, please ensure you're using buffer_string to handle both writing and reading of data.

[stravee]

Tried Dan's project real quick:

[stravee]

And this is with buffer_string, on Dan's project again:

[stravee]

If I then try buffer size+1, I get the number expected error again.

[stravee]

Btw, I'm getting this issue on 2 PCs, Win 10 and Win 11.

Could it have something to do with localization? Full points vs commas?

[DiasFranciscoA]

Could you check what is the output of the json_stringify, string_byte_length and buffer_read on your machine?

[stravee]

[stravee]

The issue also appears when I send the JSON through UDP to this machine, indicating that the issue is while parsing only?
I fixed it by simply adding apostrophes: the sensor sends "10.2" instead of 10.2

[DiasFranciscoA]

Could you please attach the json file that is saved by the buffer_write?

[stravee]

test.json

(This is from Dan's project)

(If I manually edit it and add apostrophes to the decimals, the error is gone.)

[DiasFranciscoA]

Hello,

Thank you for bringing this issue to our attention. We've identified that the problem was related to the locale settings. To address this, we've made several adjustments to how locales are set within our application:

• Numeric Locale: We've ensured the decimal separator remains consistent, specifically a period (.), which is crucial for functions like JSON parsing.
• Time Locale: This has been set to the system default to allow date and time formatting to adapt based on user settings.
• Character Type Locale: We've set this to .UTF8 to ensure all string handling functions properly support UTF-8 encoding.

These changes are expected to enhance the performance of date-to-string conversion functions and resolve the issues with both json_parse and json_decode.

Notes to QA Team

Testing Requirements:

• Date to String Functions: Thorough tests on all date-to-string to ensure they remain accurate across different locale settings.

• JSON Functionality:

  • JSON Parse: Ensure that the json_parse function correctly handles data formatted with the '.' as the decimal separator.
  • JSON Decode: Test json_decode for proper functionality to confirm that issues previously encountered are now resolved.

• String Case Conversion:

  • To Lower: Test the string_lower function to ensure that it accurately converts all applicable characters to lowercase in UTF-8.
  • To Upper: Similarly, test the string_upper function to confirm it correctly converts characters to uppercase in UTF-8.

[YYDan]

#6058 and #6048 should be checked at the same time as verifying this one.

[stravee]

By the way;

In-Game: json_parse() has issues with decimals (again)

JSON A (full stop):
{ "value": 1.30 }

Parsed value in-game: _value == 1.30

JSON B (comma):
{ "value": 1,30 }

Parsed value in-game: _value == 1

[YYDan]

@stravee Which runtime are you using and getting those results? Are you definitely on the latest Beta release?

We will double-check here, but as I showed in my comment above and is documented in more detail higher up this issue, json_parse() and json_decode() have definitely had multiple fixes recently to stop that ;)

[rwkay]

By the way;

In-Game: json_parse() has issues with decimals (again)

JSON A (full stop): { "value": 1.30 }

Parsed value in-game: _value == 1.30

JSON B (comma): { "value": 1,30 }

Parsed value in-game: _value == 1

we do not support locale specific JSON so the behaviour you are talking about there is NOT supported, we did have a bug early in 2024.6 that meant this was handled but that was a bug and not how it was intended.

[YYDan]

Ahh, yes - now I re-read Francisco's comment above, I see that's what he said at the time and that he clarified only the . decimal is going to work.

[Emc1923]

Verified working as per comments with different locale settings as of IDE v2024.600.0.584 Runtime v2024.600.0.608.

[stravee]

Ahh, yes - now I re-read Francisco's comment above, I see that's what he said at the time and that he clarified only the . decimal is going to work.

Thanks for pointing out that comment again, I had forgotten about it!

For anyone's future reference:
Since the JSON values were in string form, I simply parsed the anomalous JSON file this way and the issue was solved:

global.MillsJSON = json_parse(_json, 
function (key, value)
{
	if is_string(value) 
		value = string_replace(value, ",", ".")
	return value
});

Thanks for your hard work, guys!