Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JSONValues container for holding Python values as JSON objects if possible, and as pybind11::object otherwise #455

Merged

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Mar 13, 2024

Description

  • Add an optional unserializable_handler_fn callback to mrc::pymrc::cast_from_pyobject which will be invoked for any unsupported Python object. Allowing for serializing unsupported object.
    ex:
    pymrc::unserializable_handler_fn_t handler_fn = [](const py::object& source,
                                                       const std::string& path) {
        return nlohmann::json(py::cast<float>(source));
    };

    // decimal.Decimal is not serializable
    py::object Decimal = py::module_::import("decimal").attr("Decimal");
    py::object o       = Decimal("1.0");
    EXPECT_EQ(pymrc::cast_from_pyobject(o, handler_fn), nlohmann::json(1.0));
  • Add JSONValues container class which is an immutable container for holding Python values as nlohmann::json objects if possible, and as pybind11::object otherwise. The container can be copied and moved, but the underlying nlohmann::json object is immutable.

  • Updates nlohmann_json from 3.9 to 3.11 for patch_inplace method.

  • Incorporates ideas from @drobison00's PR Update PythonObjectCache to support get and pop operations, update utils to allow passing of cached python objects. #417 with three key differences:

    1. Changes to the cast_from_pyobject are opt-in when unserializable_handler_fn is provided, otherwise there is no behavior change to the method.
    2. Unserializable objects are stored in JSONValues rather than a global cache.
    3. Does not rely on parsing the place-holder values.

This PR is related to nv-morpheus/Morpheus#1560

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@dagardner-nv dagardner-nv self-assigned this Mar 13, 2024
@dagardner-nv dagardner-nv requested review from a team as code owners March 13, 2024 19:00
@dagardner-nv dagardner-nv added non-breaking Non-breaking change feature request New feature or request labels Mar 13, 2024
@dagardner-nv dagardner-nv marked this pull request as draft March 14, 2024 21:01
@dagardner-nv dagardner-nv marked this pull request as draft March 26, 2024 20:47
@dagardner-nv dagardner-nv marked this pull request as ready for review March 26, 2024 23:01
@dagardner-nv dagardner-nv marked this pull request as draft March 27, 2024 23:41
@dagardner-nv dagardner-nv added the skip-ci Optionally Skip CI for this PR label Mar 27, 2024
@dagardner-nv dagardner-nv removed the skip-ci Optionally Skip CI for this PR label Apr 1, 2024
@dagardner-nv dagardner-nv changed the base branch from branch-24.03 to branch-24.06 April 1, 2024 18:10
@dagardner-nv dagardner-nv marked this pull request as ready for review April 1, 2024 18:14
@dagardner-nv dagardner-nv added the conda-build Enables running the conda-build step on a PR label Apr 1, 2024
Copy link

codecov bot commented Apr 2, 2024

Codecov Report

Merging #455 (64f7459) into branch-24.06 (bd7955e) will increase coverage by 0.09%.
The diff coverage is 88.70%.

Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff                @@
##           branch-24.06     #455      +/-   ##
================================================
+ Coverage         73.68%   73.77%   +0.09%     
================================================
  Files               400      402       +2     
  Lines             14275    14395     +120     
  Branches           1113     1129      +16     
================================================
+ Hits              10518    10620     +102     
- Misses             3757     3775      +18     
Flag Coverage Δ
cpp 70.39% <94.01%> (+0.16%) ⬆️
py 41.28% <2.41%> (-0.46%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...mrc/_pymrc/include/pymrc/utilities/json_values.hpp 100.00% <100.00%> (ø)
...rc/_pymrc/include/pymrc/utilities/object_cache.hpp 0.00% <ø> (ø)
python/mrc/_pymrc/src/utils.cpp 87.50% <100.00%> (+1.95%) ⬆️
python/mrc/_pymrc/src/utilities/json_values.cpp 87.61% <87.61%> (ø)

... and 4 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd7955e...64f7459. Read the comment docs.

@mdemoret-nv
Copy link
Contributor

/merge

@rapids-bot rapids-bot bot merged commit f4e6266 into nv-morpheus:branch-24.06 Apr 3, 2024
18 checks passed
rapids-bot bot pushed a commit to nv-morpheus/Morpheus that referenced this pull request Apr 5, 2024
* Contains breaking changes for the C++ API 
* No breaking changes for the Python API
* Updates `nlohmann_json from` 3.9 to 3.11 for `patch_inplace` method.
* Requires nv-morpheus/MRC#455 to be merged first
Closes #1560

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1589
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
conda-build Enables running the conda-build step on a PR feature request New feature or request non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants