map/apply support in client/server #1504
Conversation
- added json_dumps and object_hook for python builtin functions - data_loads is not consistent for all serializations. For fastmsgpack, it uses pandas to decode to a Series; while other methods decode to a list.
- add most_formats frozen set to Serialization which includes all Serialization types except fastmsgpack. 'fastmsgpack' serializes/loads to a Series object as opposed to a list. - break apart test_map_client_server() so all Serialization types that return a list are tested in one function. 'fastmsgpack' Serialzation is tested in test_map_client_server_fastmsgpack() since its assertion has to be for all() elements.
- checks the function given to map is pandas or numpy function. Raises NotImplementedError otherwise - added simple tests for mapping numpy and pandas functions
- raise http exceptions if incoming json contains 'eval' or 'exec' which we don't support executing on server (ie 503 Forbidden) - similarly if json contains a function not in pandas/numpy namespace, raise HTTP 501 Not Implemented exception - don't limit what json_dumps can do since that is used for testing - added tests but marked as xfail since pickle serialization is currently not raising these exceptions - need to ask/update
Create `*_trusted` versions of standard `SerializationFormat`s as well.
|
@llllllllll @sandhujasmine you mind taking a once-over? |
|
I can take a look a little later today |
| from .json_dumps import json_dumps | ||
|
|
||
|
|
||
| json_dumps_trusted_ns = dict() |
There was a problem hiding this comment.
instead of the loop below this might be more clear as json_dumps_trusted_ns = jsun_dumps.funcs.copy()
There was a problem hiding this comment.
Yes, I tried that initially, but the dispatching doesn't work in that case.
There was a problem hiding this comment.
how so, iterating over the dict and updating each k/v pair should be functionally equivalent? At the end of this module would the func maps be different?
There was a problem hiding this comment.
Using json_dumps.funcs.copy() still retains the json_dumps function name, which ultimately triggers an exception in multipledispatch's inner dispatching logic; I have to do json_dumps_trusted.register(types)(func) to ensure the json_dumps_trusted function name is updated.
|
Can we move the serialization code to a new package: |
|
|
||
| from .dispatch import dispatch | ||
| from functools import partial, wraps | ||
|
|
There was a problem hiding this comment.
wraps appears to be an used import
Good idea -- done. |
kwmsmith
changed the title
|
@llllllllll @sandhujasmine I think this one is ready to merge. |
|
lgtm. Thanks! |
This picks up PR #1497 from @sandhujasmine:
*_trustedvariants forSerializationFormatinstances.object_hook.py,json_dumps.py, and their_trustedvariants created as well._trusteddispatching is implemented as a superset of their underlying untrusted counterparts to keep it DRY.