Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update json-c from 0.13.1 to 0.15 #1605

Merged
2 commits merged into from
Oct 7, 2020
Merged

Update json-c from 0.13.1 to 0.15 #1605

2 commits merged into from
Oct 7, 2020

Conversation

patrikjuvonen
Copy link
Contributor

@patrikjuvonen patrikjuvonen commented Aug 8, 2020

Summary

Speed comparison

test_json.zip

Results (250,000 toJSON + fromJSON executions, 20 times)

  • json-c 0.13.1 (MTA:SA 6941e0b client-side): 75039.3ms
  • json-c 0.13.1 (MTA:SA 6941e0b server-side): 67460.75ms
  • json-c 0.15 (MTA:SA 3afd32b client-side): 74463.05ms (0.77% improvement)
  • json-c 0.15 (MTA:SA 3afd32b server-side): 66057.75ms (2.08% improvement)

Validation

To help validate the integrity of the update I have created the following bash script that diffs between my PR branch and the official package provided from the json-c repository.

#!/bin/bash

JSONC_UPDATE_VERSION=0.15
JSONC_PATH_NAME=json-c-0.15-20200726

GIT_REPO_BRANCH=vendor/json-c-$JSONC_UPDATE_VERSION
GIT_REPO_URL=https://github.com/patrikjuvonen/mtasa-blue.git
GIT_REPO_JSONC_PATH=$GIT_DEST_DIR/vendor/json-c/

echo 1. Download and extract $JSONC_PATH_NAME...
curl -L https://github.com/json-c/json-c/archive/$JSONC_PATH_NAME.tar.gz | tar -xz --transform 's/^json-c-json-c/json-c/'

echo 2. Fetch and checkout the vendor update branch $GIT_REPO_BRANCH from $GIT_REPO_URL...
git fetch $GIT_REPO_URL $GIT_REPO_BRANCH:$GIT_REPO_BRANCH
git checkout $GIT_REPO_BRANCH

echo 3. Start checking integrity...
diff -r --strip-trailing-cr $GIT_REPO_JSONC_PATH $JSONC_PATH_NAME

echo 4. Completed.
exec $SHELL

Past json-c updates in MTA

Date From To Link
August 2018 0.12 0.13.1 (current) #269

Copy of json-c changelogs

0.15 (up to commit 870965e, 2020/07/26)

Deprecated and removed features:
--------------------------------
* Deprecate `array_list_new()` in favor of `array_list_new2()`
* Remove the THIS_FUNCTION_IS_DEPRECATED define.
* Remove config.h.win32

New features
------------
* Add a `JSON_TOKENER_ALLOW_TRAILING_CHARS` flag to allow multiple objects
  to be parsed even when `JSON_TOKENER_STRICT` is set.
* Add `json_object_new_array_ext(int)` and `array_list_new_2(int)` to allow
   arrays to be allocated with the exact size needed, when known.
* Add `json_object_array_shrink()` (and `array_list_shrink()`) and use it in 
   json_tokener to minimize the amount of memory used.
* Add a json_parse binary, for use in testing changes (not installed, but 
   available in the apps directory).

Build changes
-------------
* #639/#621 - Add symbol versions to all exported symbols
* #508/#634 - Always enable -fPIC to allow use of the json-c static library in
   other libraries
* Build both static and shared libraries at the same time.
* #626 - Restore compatibility with cmake 2.8 
* #471 - Always create directories with mode 0755, regardless of umask.
* #606/#604 - Improve support for OSes like AIX and IBM i, as well as for
   MINGW32 and old versions of MSVC
* #451/#617 - Add a DISABLE_THREAD_LOCAL_STORAGE cmake option to disable 
   the use of thread-local storage.

Significant changes and bug fixes
---------------------------------
* Split the internal json_object structure into several sub-types, one for
   each json_type (json_object_object, json_object_string, etc...).
  This improves memory usage and speed, with the benchmark under
   bench/ report 5.8% faster test time and 6%(max RSS)-12%(peak heap)
   less memory usage.
  Memory used just for json_object structures decreased 27%, so use cases
   with fewer arrays and/or strings would benefit more.
* Minimize memory usage in array handling in json_tokener by shrinking
   arrays to the exact number of elements parsed.  On bench/ benchmark:
   9% faster test time, 39%(max RSS)-50%(peak heap) less memory usage.
   Add json_object_array_shrink() and array_list_shrink() functions.
* #616 - Parsing of surrogate pairs in unicode escapes now properly handles
   incremental parsing.
* Fix incremental parsing of numbers, especially those with exponents, e.g.
   so parsing "[0", "e+", "-]" now properly returns an error.
  Strict mode now rejects missing exponents ("0e").
* Successfully return number objects at the top level even when they are
   followed by a "-", "." or "e".  This makes parsing things like "123-45"
   behave consistently with things like "123xyz".

Other changes
-------------
* #589 - Detect broken RDRAND during initialization; also, fix segfault
    in the CPUID check.
* #592 - Fix integer overflows to prevert out of bounds write on large input.
* Protect against division by zero in linkhash, when creaed with zero size.
* #602 - Fix json_parse_uint64() internal error checking, leaving the retval
    untouched in more failure cases.
* #614 - Prevent truncation when custom double formatters insert extra \0's

0.14 (up to commit 9ed00a6, 2020/04/14)

Deprecated and removed features:
--------------------------------
* bits.h has been removed
* lh_abort() has been removed
* lh_table_lookup() has been removed, use lh_table_lookup_ex() instead.
* Remove TRUE and FALSE defines, use 1 and 0 instead.

Build changes:
--------------
## Deprecated and removed features:
* bits.h has been removed
* lh_abort() has been removed
* lh_table_lookup() has been removed, use lh_table_lookup_ex() instead.
* Remove TRUE and FALSE defines, use 1 and 0 instead.
* autoconf support, including autogen.sh, has been removed.  See details about cmake, below.
* With the addition of json_tokener_get_parse_end(), access to internal fields of json_tokener, as well as use of many other symbols and types in json_tokener.h, is deprecated now.
* The use of Android.configure.mk to build for Android no longer works, and it is unknown how (or if) the new cmake-based build machinery can be used.
    * Reports of success, or pull requests to correct issues are welcome.

## Notable improvements and new features

### Builds and documentation
* Build machinery has been switched to CMake.  See README.md for details about how to build.
    * TL;DR: `mkdir build ; cd build ; cmake -DCMAKE_INSTALL_PREFIX=/some/path ../json-c ; make all test install`
    * To ease the transition, there is a `cmake-configure` wrapper that emulates the old autoconf-based configure script.
    * This has enabled improvements to the build on Windows system; also all public functions have been fixed to be properly exported.  For best results, use Visual Studio 2015 or newer.
* The json-c style guide has been updated to specify the use of clang-format, and all code has been reformatted.
    * Since many lines of code have trivial changes now, when using git blame, be sure to specify -w
* Numerous improvements have been made to the documentation including function effects on refcounts, when passing a NULL is safe, and so on.

### json_tokener changes
* Added a json_tokener_get_parse_end() function to replace direct access of tok->char_offset.
    * The char_offset field, and the rest of the json_tokener structure remain exposed for now, but expect a future release to hide it like is done with json_object_private.h
* json_tokener_parse_ex() now accepts a new JSON_TOKENER_VALIDATE_UTF8 flag to validate that input is UTF8.
    * If validation fails, json_tokener_get_error(tok) will return json_tokener_error_parse_utf8_string (see enum json_tokener_error).

### Other changes and additions
* Add support for unsigned 64-bit integers, uint64_t, to gain one extra bit of magnitude for positive ints.
    * json_tokener will now parse values up to UINT64_MAX (18446744073709551615)
    * Existing methods returning int32_t or int64_t will cap out-of-range values at INT32_MAX or INT64_MAX, preserving existing behavior.
    * The implementation includes the possibility of easily extending this to larger sizes in the future.
* A total of 7 new functions were added:
    * json_object_get_uint64 ( struct json_object const* jso )
    * json_object_new_uint64 ( uint64_t i )
    * json_object_set_uint64 ( struct json_object* jso, uint64_t new_value )
    * json_parse_uint64 ( char const* buf, uint64_t* retval )
        * See description of uint64 support, above.
    * json_tokener_get_parse_end ( struct json_tokener* tok )
        * See details under "json_tokener changes", above.
    * json_object_from_fd_ex ( int fd, int in_depth )
        * Allows the max nesting depth to be specified.
    * json_object_new_null ( )
        * Simply returns NULL.  Its use is not recommended.
* The size of struct json_object has decreased from 96 bytes to 88 bytes.

### Testing
* Many updates were made to test cases, increasing code coverage.
* There is now a quick way (JSONC_TEST_TRACE=1) to turn on shell tracing in tests.
* To run tests, use `make test`; the old "check" target no longer exists.

## Significant bug fixes
For the full list of issues and pull requests since the previous release, please see issues_closed_for_0.14.md

* [Issue #389](https://github.com/json-c/json-c/issues/389): Add an assert to explicitly crash when _ref_count is corrupted, instead of a later "double free" error.
* [Issue #407](https://github.com/json-c/json-c/issues/407): fix incorrect casts in calls to ctype functions (isdigit and isspace) so we don't crash when asserts are enabled on certain platforms and characters > 128 are parsed.
* [Issue #418](https://github.com/json-c/json-c/issues/418): Fix docs for json_util_from_fd and json_util_from_file to say that they return NULL on failures.
* [Issue #422](https://github.com/json-c/json-c/issues/422): json_object.c:set errno in json_object_get_double() when called on a json_type_string object with bad content.
* [Issue #453](https://github.com/json-c/json-c/issues/453): Fixed misalignment in JSON serialization when JSON_C_TO_STRING_SPACED and JSON_C_TO_STRING_PRETTY are used together.
* [Issue #463](https://github.com/json-c/json-c/issues/463): fix newlocale() call to use LC_NUMERIC_MASK instead of LC_NUMERIC, and remove incorrect comment.
* [Issue #486](https://github.com/json-c/json-c/issues/486): append a missing ".0" to negative double values to ensure they are serialized as floating point numbers.
* [Issue #488](https://github.com/json-c/json-c/issues/488): use JSON_EXPORT on functions so they are properly exported on Windows.
* [Issue #539](https://github.com/json-c/json-c/issues/539): use an internal-only serializer function in json_object_new_double_s() to avoid potential conflicts with user code that uses the json_object_userdata_to_json_string serializer.

@patrikjuvonen patrikjuvonen added enhancement New feature or request upstream Related to vendor library labels Aug 8, 2020
@patrikjuvonen patrikjuvonen added this to the Backlog milestone Aug 8, 2020
@patrikjuvonen patrikjuvonen added this to In progress in Vendor upgrades via automation Aug 8, 2020
@Dutchman101 Dutchman101 self-requested a review October 7, 2020 03:27
Copy link
Member

@Dutchman101 Dutchman101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest we merge this soon, so it can get atleast 2 weeks of testing while we work through #1645's checkklist, and then be included in the release of 1.5.8

@Pirulax
Copy link
Contributor

Pirulax commented Oct 7, 2020

I think you should've excluded the lua array allocation from the test, to minimize the amount of variables.
Nevertheless, im just being nitpicky, its fine

@ghost ghost merged commit 288a899 into multitheftauto:master Oct 7, 2020
Vendor upgrades automation moved this from In progress to Done Oct 7, 2020
@ghost ghost modified the milestones: Backlog, 1.5.8 Oct 7, 2020
@patrikjuvonen patrikjuvonen deleted the vendor/json-c-0.15 branch October 9, 2020 21:20
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request upstream Related to vendor library
Projects
Development

Successfully merging this pull request may close these issues.

None yet

3 participants