Skip to content

Implement recommended fixes from OSTIF audit#313

Merged
etrepum merged 7 commits into
masterfrom
audit-fixes
Apr 6, 2023
Merged

Implement recommended fixes from OSTIF audit#313
etrepum merged 7 commits into
masterfrom
audit-fixes

Conversation

@etrepum

@etrepum etrepum commented Apr 4, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Implement security hardening measures based on a source code audit of simplejson 3.18.4 by X41 D-Sec GmbH and sponsored by the OSTIF.

  • Fix invalid handling of unicode escape sequences in the pure Python
    implementation of the decoder (SJ-PT-23-01)
  • Fix missing reference count decrease if PyOS_string_to_double raises
    an exception in Python 2.x; was probably unreachable (SJ-PT-23-02)
  • Backport the integer string length limitation from Python 3.11 to
    limit quadratic number parsing (SJ-PT-23-03)
  • Fix inconsistencies with error messages between the C and Python
    implementations (SJ-PT-23-100)
  • Remove unused unichr import from encoder (SJ-PT-23-101)
  • Remove unused namedtuple_as_object and tuple_as_array arguments from
    simplejson.load (SJ-PT-23-102)
  • Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
  • Change default of allow_nan from True to False and add allow_nan
    to decoder (SJ-PT-23-107)

Several suggested improvements were not implemented in this release and will be considered in the future:

  • SJ-PT-23-104: Type Hints Not Used - Implementing type hints with annotations is not possible for as long as Python 2 is supported. Using stub files or revisiting this in the future when Python 2 support is removed will be considered in a later release.
  • SJ-PT-23-105: Deprecated Python Versions Supported - Without a way to get usage metrics, it's hard to say how many people are still using recent versions of simplejson and Python 2, so I would prefer to maintain support for a while longer.
  • SJ-PT-23-108: Support of Duplicate Key Names - I haven't had a lot of requests to provide this feature, and anyone looking to do this can implement it with object_pairs_hook. The default behavior of "last key wins" is consistent with JavaScript's JSON implementation.
  • SJ-PT-23-106: Unsigned Git Commits - PR merges were already verified since I update them with the GitHub UX but I will also start signing my commits with an SSH key. I have also enabled a tag protection rule for *. I did not enable a branch protection rule to require all commits to be signed since that would prevent accepting third party contributions without first rebasing myself.

The full public report is available here: https://www.x41-dsec.de/static/reports/X41-OSTIF-simplejson-CodeRview-2023-04-18.pdf
See also:

etrepum added 7 commits April 4, 2023 09:59
@etrepum
SJ-PT-23-01: Fix invalid handling of unicode escape sequences in Pyth…
7c2ccb7 
…on decoder
@etrepum
SJ-PT-23-02: Fix missing reference count decrease
1e495c1
@etrepum
SJ-PT-23-03: Backport integer string length limitation to limit quadr…
59dac4e 
…atic parsing
@etrepum
SJ-PT-23-100: Fix inconsistencies in error messages between C and Pyt…
dbd0aa3 
…hon implementations
@etrepum
SJ-PT-23-101: Remove unused unichr import from encoder
440a5e4
@etrepum
Additional security hardening improvements:
2cbc419 
* Remove unused namedtuple_as_object and tuple_as_array arguments from
  simplejson.load (SJ-PT-23-102)
* Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
* Change default of allow_nan from True to False and add allow_nan
  to decoder (SJ-PT-23-107)
@etrepum
Update CHANGES.txt
ec4a3d5
@etrepum etrepum enabled auto-merge April 6, 2023 16:53
@etrepum etrepum merged commit 1a4995d into master Apr 6, 2023
This was referenced Apr 6, 2023
Merged
Merged
Merged
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 7, 2023
py-simplejson: update to 3.19.1
e2fe0e1 
Version 3.19.1 released 2023-04-06

* This release contains security hardening measures based on recommendations
  by a security audit sponsored by OSTIF and conducted by X41 D-Sec GmbH.
  Several of these measures include changing defaults to be more strict,
  by default simplejson will now only consume and produce compliant JSON,
  but the flags still exist for any backwards compatibility needs.
  No high priority issues were discovered, the reference count
  leak is thought to be unreachable since the digits of the float are
  checked before PyOS_string_to_double is called.
  A link to the public version of this report will be included in a
  future release of simplejson. The following fixes were implemented in
  one PR: simplejson/simplejson#313
* Fix invalid handling of unicode escape sequences in the pure Python
  implementation of the decoder (SJ-PT-23-01)
* Fix missing reference count decrease if PyOS_string_to_double raises
  an exception in Python 2.x; was probably unreachable (SJ-PT-23-02)
* Backport the integer string length limitation from Python 3.11 to
  limit quadratic number parsing (SJ-PT-23-03)
* Fix inconsistencies with error messages between the C and Python
  implementations (SJ-PT-23-100)
* Remove unused unichr import from encoder (SJ-PT-23-101)
* Remove unused namedtuple_as_object and tuple_as_array arguments from
  simplejson.load (SJ-PT-23-102)
* Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
* Change default of allow_nan from True to False and add allow_nan
  to decoder (SJ-PT-23-107)
This was referenced Apr 8, 2023
Merged
Open
kraj pushed a commit to YoeDistro/meta-openembedded that referenced this pull request Apr 10, 2023
@wangmingyu84 @kraj
python3-simplejson: upgrade 3.18.4 -> 3.19.1
3c49711 
Changelog:
============
* This release contains security hardening measures based on recommendations
  by a security audit sponsored by OSTIF and conducted by X41 D-Sec GmbH.
  Several of these measures include changing defaults to be more strict,
  by default simplejson will now only consume and produce compliant JSON,
  but the flags still exist for any backwards compatibility needs.
  No high priority issues were discovered, the reference count
  leak is thought to be unreachable since the digits of the float are
  checked before PyOS_string_to_double is called.
  A link to the public version of this report will be included in a
  future release of simplejson. The following fixes were implemented in
  one PR: simplejson/simplejson#313
* Fix invalid handling of unicode escape sequences in the pure Python
  implementation of the decoder (SJ-PT-23-01)
* Fix missing reference count decrease if PyOS_string_to_double raises
  an exception in Python 2.x; was probably unreachable (SJ-PT-23-02)
* Backport the integer string length limitation from Python 3.11 to
  limit quadratic number parsing (SJ-PT-23-03)
* Fix inconsistencies with error messages between the C and Python
  implementations (SJ-PT-23-100)
* Remove unused unichr import from encoder (SJ-PT-23-101)
* Remove unused namedtuple_as_object and tuple_as_array arguments from
  simplejson.load (SJ-PT-23-102)
* Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
* Change default of allow_nan from True to False and add allow_nan
  to decoder (SJ-PT-23-107)

Signed-off-by: Wang Mingyu <wangmy@fujitsu.com>
Signed-off-by: Khem Raj <raj.khem@gmail.com>
This was referenced May 31, 2023
Open
Merged
Merged
@renovate renovate Bot mentioned this pull request Jul 8, 2023
Merged
1 task
@winterbot winterbot mentioned this pull request Aug 14, 2023
Merged
1 task
@renovate renovate Bot mentioned this pull request Aug 19, 2023
Closed
1 task
@renovate renovate Bot mentioned this pull request Aug 31, 2023
Merged
1 task
@renovate renovate Bot mentioned this pull request Oct 23, 2023
Closed
1 task
@renovate renovate Bot mentioned this pull request Jun 19, 2024
Merged
1 task
@etrepum etrepum deleted the audit-fixes branch April 12, 2026 20:31
msk pushed a commit to msk/pkgsrc that referenced this pull request May 11, 2026
py-simplejson: update to 3.19.1
84b5c95 
Version 3.19.1 released 2023-04-06

* This release contains security hardening measures based on recommendations
  by a security audit sponsored by OSTIF and conducted by X41 D-Sec GmbH.
  Several of these measures include changing defaults to be more strict,
  by default simplejson will now only consume and produce compliant JSON,
  but the flags still exist for any backwards compatibility needs.
  No high priority issues were discovered, the reference count
  leak is thought to be unreachable since the digits of the float are
  checked before PyOS_string_to_double is called.
  A link to the public version of this report will be included in a
  future release of simplejson. The following fixes were implemented in
  one PR: simplejson/simplejson#313
* Fix invalid handling of unicode escape sequences in the pure Python
  implementation of the decoder (SJ-PT-23-01)
* Fix missing reference count decrease if PyOS_string_to_double raises
  an exception in Python 2.x; was probably unreachable (SJ-PT-23-02)
* Backport the integer string length limitation from Python 3.11 to
  limit quadratic number parsing (SJ-PT-23-03)
* Fix inconsistencies with error messages between the C and Python
  implementations (SJ-PT-23-100)
* Remove unused unichr import from encoder (SJ-PT-23-101)
* Remove unused namedtuple_as_object and tuple_as_array arguments from
  simplejson.load (SJ-PT-23-102)
* Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
* Change default of allow_nan from True to False and add allow_nan
  to decoder (SJ-PT-23-107)
jperkin pushed a commit to TritonDataCenter/pkgsrc that referenced this pull request May 14, 2026
py-simplejson: update to 3.19.1
896f476 
Version 3.19.1 released 2023-04-06

* This release contains security hardening measures based on recommendations
  by a security audit sponsored by OSTIF and conducted by X41 D-Sec GmbH.
  Several of these measures include changing defaults to be more strict,
  by default simplejson will now only consume and produce compliant JSON,
  but the flags still exist for any backwards compatibility needs.
  No high priority issues were discovered, the reference count
  leak is thought to be unreachable since the digits of the float are
  checked before PyOS_string_to_double is called.
  A link to the public version of this report will be included in a
  future release of simplejson. The following fixes were implemented in
  one PR: simplejson/simplejson#313
* Fix invalid handling of unicode escape sequences in the pure Python
  implementation of the decoder (SJ-PT-23-01)
* Fix missing reference count decrease if PyOS_string_to_double raises
  an exception in Python 2.x; was probably unreachable (SJ-PT-23-02)
* Backport the integer string length limitation from Python 3.11 to
  limit quadratic number parsing (SJ-PT-23-03)
* Fix inconsistencies with error messages between the C and Python
  implementations (SJ-PT-23-100)
* Remove unused unichr import from encoder (SJ-PT-23-101)
* Remove unused namedtuple_as_object and tuple_as_array arguments from
  simplejson.load (SJ-PT-23-102)
* Remove vestigial _one_shot code from iterencode (SJ-PT-23-103)
* Change default of allow_nan from True to False and add allow_nan
  to decoder (SJ-PT-23-107)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@etrepum