Full JSON/YAML support for Param

[jbednar]

Param's JSON support is currently quite limited. It can serialize and deserialize all the basic Parameter types, including types matching the native JSON/JS types (Integer, Float, String, etc.), containers of such types (List, Dict, etc.), or Python/Param-specific types that need to be stored in a special format and then decoded back into Python types (Tuple, Array, DataFrame, etc.). However, there is not currently any support for storing or restoring:

• the Parameterized type of the object being serialized (which the deserialization code currently needs to know beforehand)
• nested Parameterized objects (Parameter values that are themselves Parameterized objects)
• directly to or from YAML (which is a superset of JSON)

These limitations prevent this form of text serialization from being used as a general-purpose human-readable text format for specifying collections of Parameterized objects. Ideally, we'd be able to serialize a large and comprehensive class of objects to pass around as JSON (in a server context), and would be able to accept a hierarchical YAML file instantiating various Parameterized classes, each of which may contain arbitrarily nested other Parameterized objects also specified. Of course, JSON itself was not designed to represent Parameterized objects or even Python objects, nor does it include explicit mechanisms for extending the language (unlike XML). Possible ways around this:

1. YAML does include custom-type support using !, and one approach is to support additional functionality when using YAML only (as described for e.g. pyyaml) that accepts a specified custom type and instantiates it. E.g.:

someobj:
  !custompackage.custommodule.CustomType 
    name: "My object"
    size: 7
    nestedobj:
      !custompackage.custommodule.CustomType2:
        name: "Nested object"
        desc: "str"

2. Establish a convention using JSON (and thus accepted in YAML), e.g. using a dict containing a special keyword like __type that you hope is unlikely to occur by chance. For example, in JSON:

{"someobj": 
    {"name":"My object", "__type":"custompackage.custommodule.CustomType", "size": 7, "nestedobj": 
        {"name": "Nested object", "__type":"custompackage.custommodule.CustomType2", "desc": "str""}}}

or in YAML:

someobj: 
  name: "My object"
  __type: "custompackage.custommodule.CustomType"
  size: 7
  nestedobj:
    name: "Nested object"
    __type: "custompackage.custommodule.CustomType2"
    desc: "str"

(I could easily have gotten any of the syntax here confused, so please update it if you spot an error!)

I think the pros of option 1 are that it fully follows the YAML spec and is thus well defined with no reliance on heuristics, and that it's clear to a non-Python recipient of such a file that it requires some custom handling to parse the ! entries. The cons are that it won't be supported for JSON output.

The pros of option 2 are that it's supported for both JSON and YAML equivalently, and the cons would be that it relies on a heuristic that won't necessarily be evident to some non-Python code that tries to parse such objects, potentially leading to undetected errors and missing functionality.

Anyone have an option 3? I think we should go ahead with some option, in any case.

[jbednar]

This issue relates to a variety of previous issues and PRs that are worth reading before making decisions:

• Current JSON support implementation: JSON serialization and schema generation #414
• Request for serialization to text files: Feature Request: Serialization for other file types #485
• Earlier proposal for YAML support, written before the current JSON support was added (with extensive proposed YAML example syntax alternatives): Add json/yaml serialization #159
• Earlier proposal that led to the current JSON literal serialization support: Literal serialization and deserialization #382
• Support for deserializing from a text file: Support for deserialization of file types into Array, DataFrame #482 (open PR)
• Earlier PR for JSON/YAML support, not merged: Serialize / deserialize param objects: yaml and json #153

[sdrobert]

Hello, occasional busybody here.

If the first concern can be addressed (determining the outermost parameterized type), the second concern shouldn't be an issue, right? You should be able to infer the appropriate parameterized class from the outermost class' member.

Personally I have very little stake in the matter because I can't imagine a scenario where I need to deserialize something which I know how to manipulate but not the class it belongs to. That said, I would vote for option 2 because a) there's really no other way around JSON; and b) option 1 will fail if the class is not available (no-one outside of the param ecosystem can manipulate it).

Thanks for your time,
Sean

[jbednar]

If the first concern can be addressed (determining the outermost parameterized type), the second concern shouldn't be an issue, right? You should be able to infer the appropriate parameterized class from the outermost class' member.

Right; those first two concerns should be solved at once, if the type is stored along with the set of parameters; the same representation should work both for the outermost class and for nested classes.

Personally I have very little stake in the matter because I can't imagine a scenario where I need to deserialize something which I know how to manipulate but not the class it belongs to.

There are a lot of places this comes up:

• Even if exporting and restoring only a single class, having that class declared can be an important validation, and allows detecting bogus files provided as arguments.
• Even if you already know the class of what you are getting, users should be allowed to provide a subclass. E.g. you could always expect a Simulation, but you might be provided an EventDrivenSimulation, a TimebasedSimulation, etc., and you'd want to instantiate the correct class in each case so that you can handle all the parameters associated with the subclasses, not just the superclass.
• Even if you know precisely which class you should be getting, if any of the parameters of that class can themselves be a Parameterized object (e.g. a Scene that might contain a Box with a Shape inside), you have to know the classes for all the subobjects in order to instantiate any of it.

That said, I would vote for option 2 because a) there's really no other way around JSON; and b) option 1 will fail if the class is not available (no-one outside of the param ecosystem can manipulate it).

I agree, though I would imagine that e.g. JavaScript could define some sort of handler for !!python objects...

[jlstevens]

I definitely do not like any approach that relies on YAML features not supported by JSON. Out of the two options presented above, I prefer the second.

That said, sticking the qualified path to the parameterized into the dictionary of parameters is only one approach. What is necessary is to associate the collection of parameters to the class and this could be done in two other obvious ways that map to JSON:

1. A tuple. The simplest case would be something like ('custompackage.custommodule.CustomType', {'size':7})
2. An outer dictionary which would be something like {'parameterized':'custompackage.custommodule.CustomType', 'parameters':{'size':7}}

Here I prefer the second option as the tuple is shorter but needlessly encodes semantics by position. In addition, I would want to support an extension of the second option to allow args and kwargs so you can reconstruct the object:

{'parameterized':'custompackage.custommodule.CustomType', 'parameters':{'size':7}, 'args':[33, 45], 'kwargs':{'foo':99}}

Of course supporting args and kwargs would need a mechanism that hooks into param but is supplied by the class (e.g. with some special method).

Note that we now have three different possible JSON schemas for our object depending on which approach is chosen. I think this explains why the current schema method needs some outer padding to a 'full' schema: the form of that outer padding depends on the convention used. I am in favor of my last suggestion as you no longer have any possibility of a namespace clash, it is explicit and it is extensible.

[jbednar]

@jlstevens , how does your proposed tuple support nested objects? E.g. if the value of a param.Parameter is ('custompackage.custommodule.CustomType', {'size':7}), how can we know that should be instantiated as CustomType(size=7), as opposed to the literal tuple (str, dict) specified? Having a heuristic special value like __type (or some even more obscure name) is designed to make it possible to detect that it's not simply an ordinary literal. Presumably for your dict proposal, 'parameterized' could be '__parameterized' or something obscure that achieves the same thing. I guess for the tuple it could be ('__parameterized', 'custompackage.custommodule.CustomType', {'size':7})?

With any of these approaches, I think it's very important to consider how it would be expressed in YAML. One of the main uses for such serialization is for a human being to be able to write a reasonable, clear, concise, and compact YAML specification, and I'd like to avoid adding extra levels of nesting that are not typically needed. I agree it's nice to have a place for args and kwargs, but I'm not sure that justifies an extra nesting level as I think there are other ways to handle those.

[jlstevens]

Because JSON doesn't have the concept of classes, all of these proposals will always be heuristics. For every scheme above there is an object that will be understood to be a class by param when in fact it isn't.

Your question indicates another reason I prefer the dictionary format to the tuple format. If a dictionary has a key called parameterized and that fully qualified path points to a real parameterized class in the namespace, then the intention is pretty likely to be that it should be turned into that parameterized object. Of course that might not be the case, but it seems unlikely that you would stumble on that by accident (as unlikely as using __type name for instance and I don't think __parameterized helps much here).

With any of these approaches, I think it's very important to consider how it would be expressed in YAML.

Agreed which is why I would like to avoid the horrible names starting with one or more underscores.

I agree it's nice to have a place for args and kwargs, but I'm not sure that justifies an extra nesting level as I think there are other ways to handle those.

I think the extra level nesting is better (clear and explicit) even ignoring args and kwargs. That those could also be supported is a nice additional bonus imho.

[jlstevens]

Note the args and kwargs options would be optional even if supported. An object's YAML could be something like:

parameterized: "custompackage.custommodule.CustomType"
parameters:
  name: 'my-object'
  size: 7

In practice this would typically be put under a key in YAML but that is not part of the yaml param needs to process to build the object (and not the YAML param would necessarily generate for you):

someobj:
  parameterized: "custompackage.custommodule.CustomType"
  parameters:
    name: 'my-object'
    size: 7

I considered whether the name parameter could be used for the outer key (dropping it from the parameter list) but thinking about this more, this approach won't really work consistently for nested parameters where the parameter name may have no relation to the name of the nested parameterized object. In other words, the name someobj has to come from the user and is not something param can generate for you automatically.

I also think this is a reasonable amount of nesting: deeply nested parameterized objects are going to be horrible no matter what so I think we should not consider what nesting looks like too much and only make sure it would be supported.

[jbednar]

For a nested Parameterized, the someobj name would be the name of the Parameter whose value is a Parameterized, e.g. nestedobj above. We could also require such a name at the top level, but at that point it just declares the Python attribute name of the whole object you're returning, and usually you wouldn't want to put some arbitrary object in the global or some other namespace (e.g. topo.sim:); instead the reader would know where they want the result to go already and only wants the value, not the name. So I don't think it makes sense to require a name at the top level, as it will just be ignored, right?

I don't think it's safe to ignore nested objects, because the main reason to use Param for a configuration file compared to e.g. TOML or INI is the power of compositionality. For your proposal I think the nesting would look like:

someobj:
  parameterized: "custompackage.custommodule.CustomType"
  parameters:
    name: 'my-object'
    size: 7
    nestedobj: 
      name: "Nested object"
      parameterized: "custompackage.custommodule.CustomType2"
      parameters:
        desc: "str"