Add "object" variables (#118)

* add "object" variable

* rename "object" to "any_object"

object is already a reserved word in Python, may be confusing to use it
here.

* handle object variable everywhere

* add tests

* update docs

* update release notes

doc/api.rst

@@ -154,6 +154,7 @@ Variable
 
    variable
    index
+   any_object
    foreign
    group
    on_demand

doc/framework.rst

@@ -134,26 +134,24 @@ independently of each other.
 Group variables
 ~~~~~~~~~~~~~~~
 
-In some cases, using group variables may provide an elegant
-alternative to hard-coded links between processes.
+In some cases, using group variables may provide an elegant alternative to
+hard-coded links between processes.
 
 The membership of variables to one or several groups is defined via their
-``groups`` attribute. If you want to use in a separate process all the variables
-of a group, instead of explicitly declaring foreign variables you can declare a
-:func:`~xsimlab.group` variable. The latter behaves like an iterable of foreign
-variables pointing to each of the variables that are members of the group,
-across the model.
-
-Note that group variables only support ``intent='in'``, i.e, group
-variables should only be used to get the values of multiple foreign
-variables of a same group.
-
-Group variables are useful particularly in cases where you want to
-combine (aggregate) different processes that act on the same
-variable, e.g. in landscape evolution modeling combine the effect of
-different erosion processes on the evolution of the surface
-elevation. This way you can easily add or remove processes to/from a
-model and avoid missing or broken links between processes.
+``groups`` attribute. If you want to reuse in a separate process all the
+variables of a given group, instead of explicitly declaring each of them as
+foreign variables you can simply declare a :func:`~xsimlab.group` variable. The
+latter behaves like an iterable of foreign variables pointing to each of the
+variables (model-wise) that are members of the same group.
+
+Note that group variables implicitly have ``intent='in'``, i.e, they could only
+be used to get the values of multiple foreign variables, not set their values.
+
+Group variables are useful particularly in cases where you want to combine
+(aggregate) different processes that act on the same variable, e.g. in landscape
+evolution modeling combine the effect of different erosion processes on the
+evolution of the surface elevation. This way you can easily add or remove
+processes to/from a model and avoid missing or broken links between processes.
 
 On-demand variables
 ~~~~~~~~~~~~~~~~~~~
@@ -165,7 +163,7 @@ given few times (or not at all). These are declared using
 :func:`~xsimlab.on_demand` and must implement in the same
 process-ified class a dedicated method -- i.e., decorated with
 ``@foo.compute`` where ``foo`` is the name of the variable -- that
-returns their value. They have always ``intent='out'``.
+returns their value. They implicitly have ``intent='out'``.
 
 On-demand variables are useful, e.g., for optional model diagnostics.
 
@@ -174,8 +172,24 @@ Index variables
 
 Index variables are intended for indexing data of other variables in a model
 like, e.g., coordinate labels of grid nodes. They are declared using
-:func:`~xsimlab.index`. They have always ``intent='out'`` although their values
-could be computed from other input variables.
+:func:`~xsimlab.index`. They implicitly have ``intent='out'``, although their
+values could be computed from other input variables.
+
+'Object' variables
+~~~~~~~~~~~~~~~~~~
+
+Sometimes we need to share between processes one or more arbitrary objects,
+e.g., callables or instances of custom classes that have no array-like
+interface. Those objects should be declared in process-decorated classes using
+:func:`~xsimlab.any_object`.
+
+Within a model, those 'object' variables are reserved for internal use only,
+i.e., they never require an input value (they implicitly have ``intent='out'``)
+and they can't be saved as outputs as their value may not be compatible with the
+xarray data model. Of course, it is still possible to create those objects using
+data from other (input) variables declared in the process. Likewise, their data
+could still be coerced into a scalar or an array and be saved as output via
+another variable.
 
 Simulation workflow
 -------------------

doc/whats_new.rst

@@ -76,6 +76,8 @@ Enhancements
   (:issue:`104`, :issue:`110`).
 - Added the ability to easily run batches of simulations using the ``batch_dim``
   parameter of :func:`xarray.Dataset.xsimlab.run` (:issue:`115`).
+- Added 'object' variables :func:`~xsimlab.any_object` for sharing arbitrary
+  Python objects between processes (:issue:`118`).
 
 Bug fixes
 ~~~~~~~~~

xsimlab/__init__.py

@@ -13,7 +13,7 @@
     runtime,
     variable_info,
 )
-from .variable import variable, index, on_demand, foreign, group
+from .variable import any_object, variable, index, on_demand, foreign, group
 from .xr_accessor import SimlabAccessor, create_setup
 from . import monitoring
 

xsimlab/formatting.py

@@ -64,6 +64,9 @@ def _summarize_var(var, process, col_width):
 
         var_info = f"{link_symbol} {'.'.join(key)}"
 
+    elif var_type == VarType.OBJECT:
+        var_info = var.metadata["description"]
+
     else:
         var_dims = " or ".join([str(d) for d in var.metadata["dims"]])
 

xsimlab/model.py

@@ -30,6 +30,23 @@ def _flatten_keys(key_seq):
     return flat_keys
 
 
+def get_model_variables(p_mapping, **kwargs):
+    """Get variables in the model (processes mapping) as a list of
+    ``(process_name, var_name)`` tuples.
+
+    **kwargs may be used to return only a subset of the variables.
+
+    """
+    var_keys = []
+
+    for p_name, proc in p_mapping.items():
+        var_keys += [
+            (p_name, var_name) for var_name in filter_variables(proc, **kwargs)
+        ]
+
+    return var_keys
+
+
 class _ModelBuilder:
     """Used to iteratively build a new model.
 
@@ -129,7 +146,7 @@ def _get_var_key(self, p_name, var):
 
         var_type = var.metadata["var_type"]
 
-        if var_type in (VarType.VARIABLE, VarType.INDEX):
+        if var_type in (VarType.VARIABLE, VarType.INDEX, VarType.OBJECT):
             state_key = (p_name, var.name)
 
         elif var_type == VarType.ON_DEMAND:
@@ -242,20 +259,7 @@ def filter_out(var):
             raise ValueError(f"Conflict(s) found in given variable intents:\n{msg}")
 
     def get_variables(self, **kwargs):
-        """Get variables in the model as a list of
-        ``(process_name, var_name)`` tuples.
-
-        **kwargs may be used to return only a subset of the variables.
-
-        """
-        all_keys = []
-
-        for p_name, p_cls in self._processes_cls.items():
-            all_keys += [
-                (p_name, var_name) for var_name in filter_variables(p_cls, **kwargs)
-            ]
-
-        return all_keys
+        return get_model_variables(self._processes_cls, **kwargs)
 
     def get_input_variables(self):
         """Get all input variables in the model as a list of

xsimlab/process.py

@@ -429,6 +429,7 @@ class _ProcessBuilder:
     _make_prop_funcs = {
         VarType.VARIABLE: _make_property_variable,
         VarType.INDEX: _make_property_variable,
+        VarType.OBJECT: _make_property_variable,
         VarType.ON_DEMAND: _make_property_on_demand,
         VarType.FOREIGN: _make_property_variable,
         VarType.GROUP: _make_property_group,

xsimlab/tests/fixture_process.py

@@ -35,6 +35,7 @@ class ExampleProcess:
     out_var = xs.variable(groups="example_group", intent="out")
     inout_var = xs.variable(intent="inout", converter=int)
     od_var = xs.on_demand()
+    obj_var = xs.any_object(description="arbitrary object")
 
     in_foreign_var = xs.foreign(SomeProcess, "some_var")
     in_foreign_var2 = xs.foreign(AnotherProcess, "some_var")
@@ -66,6 +67,7 @@ def example_process_repr():
         out_var              [out]
         inout_var          [inout]
         od_var               [out]
+        obj_var              [out] arbitrary object
         in_foreign_var        [in] <--- SomeProcess.some_var
         in_foreign_var2       [in] <--- AnotherProcess.some_var
         out_foreign_var      [out] ---> AnotherProcess.another_var
@@ -139,6 +141,7 @@ def __init__(self):
             "in_var": ("example_process", "in_var"),
             "out_var": ("example_process", "out_var"),
             "inout_var": ("example_process", "inout_var"),
+            "obj_var": ("example_process", "obj_var"),
             "in_foreign_var": ("some_process", "some_var"),
             "in_foreign_var2": ("some_process", "some_var"),
             "out_foreign_var": ("another_process", "another_var"),
@@ -171,6 +174,7 @@ def example_process_in_model_repr():
         out_var              [out]
         inout_var          [inout]
         od_var               [out]
+        obj_var              [out] arbitrary object
         in_foreign_var        [in] <--- some_process.some_var
         in_foreign_var2       [in] <--- some_process.some_var
         out_foreign_var      [out] ---> another_process.another_var

xsimlab/tests/test_model.py

@@ -4,7 +4,15 @@
 
 import xsimlab as xs
 from xsimlab.process import get_process_cls
+from xsimlab.model import get_model_variables
 from xsimlab.tests.fixture_model import AddOnDemand, InitProfile, Profile
+from xsimlab.variable import VarType
+
+
+def test_get_model_variables(model):
+    idx_vars = get_model_variables(model, var_type=VarType.INDEX)
+
+    assert idx_vars == model.index_vars
 
 
 class TestModelBuilder:
@@ -75,6 +83,15 @@ def test_set_process_keys(
         assert actual_state_keys == expected_state_keys
         assert actual_od_keys == expected_od_keys
 
+    def test_object_variable(self):
+        @xs.process
+        class P:
+            obj = xs.any_object()
+
+        m = xs.Model({"p": P})
+
+        assert m.p.__xsimlab_state_keys__["obj"] == ("p", "obj")
+
     def test_multiple_groups(self):
         @xs.process
         class A:

xsimlab/tests/test_process.py

@@ -59,6 +59,7 @@ class NotAProcess:
                 "in_foreign_od_var",
                 "group_var",
                 "od_var",
+                "obj_var",
             },
         ),
         ({"var_type": "variable"}, {"in_var", "out_var", "inout_var"}),
@@ -72,7 +73,7 @@ class NotAProcess:
                 "group_var",
             },
         ),
-        ({"intent": "out"}, {"out_var", "out_foreign_var", "od_var"}),
+        ({"intent": "out"}, {"out_var", "out_foreign_var", "od_var", "obj_var"}),
         ({"group": "example_group"}, {"out_var"}),
         (
             {
@@ -143,30 +144,24 @@ def test_process_properties_readonly(cls, var_name, prop_is_read_only):
 
 
 def test_process_properties_errors():
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(ValueError, match=r".*links to group variable.*"):
 
         @xs.process
         class Process1:
             invalid_var = xs.foreign(ExampleProcess, "group_var")
 
-    assert "links to group variable" in str(excinfo.value)
-
-    with pytest.raises(ValueError) as excinfo:
+    with pytest.raises(ValueError, match=r".*both have intent='out'.*"):
 
         @xs.process
         class Process2:
             invalid_var = xs.foreign(ExampleProcess, "out_var", intent="out")
 
-    assert "both have intent='out'" in str(excinfo.value)
-
-    with pytest.raises(KeyError) as excinfo:
+    with pytest.raises(KeyError, match=r"No compute method found.*"):
 
         @xs.process
         class Process3:
             var = xs.on_demand()
 
-    assert "No compute method found" in str(excinfo.value)
-
 
 def test_process_properties_docstrings(in_var_details):
     # order of lines in string is not ensured (printed from a dictionary)
@@ -186,6 +181,9 @@ def test_process_properties_values(processes_with_state):
     example_process.inout_var = 2
     assert example_process.inout_var == 2
 
+    example_process.obj_var = lambda x: x * 2
+    assert example_process.obj_var(2) == 4
+
     example_process.out_foreign_var = 3
     assert another_process.another_var == 3
 

xsimlab/tests/test_xr_accessor.py

@@ -320,6 +320,17 @@ def test_set_output_vars(self, model):
         with pytest.warns(FutureWarning):
             ds.xsimlab._set_output_vars(model, {"out": ("profile", "u_opp")})
 
+    def test_set_output_object_vars(self):
+        @xs.process
+        class P:
+            obj = xs.any_object()
+
+        m = xs.Model({"p": P})
+        ds = xr.Dataset()
+
+        with pytest.raises(ValueError, match=r"Object variables can't be set.*"):
+            ds.xsimlab._set_output_vars(m, {("p", "obj"): None})
+
     def test_output_vars(self, model):
         o_vars = {
             ("profile", "u_opp"): None,

xsimlab/variable.py

@@ -12,6 +12,7 @@ class VarType(Enum):
     VARIABLE = "variable"
     INDEX = "index"
     ON_DEMAND = "on_demand"
+    OBJECT = "object"
     FOREIGN = "foreign"
     GROUP = "group"
 
@@ -335,6 +336,39 @@ def on_demand(
     return attr.attrib(metadata=metadata, init=False, repr=False)
 
 
+def any_object(groups=None, description="", attrs=None):
+    """Create a variable used to hold any arbitrary object that needs to be shared
+    with other process classes.
+
+    Use this instead of :func:`~xsimlab.variable` if you need to pass anything
+    other than scalar/array values to other processes (e.g., a callable or an
+    instance of a custom class that have no array-like interface).
+
+    Unlike regular variables, 'object' variables are not intended to be used as
+    model inputs or outputs. Additionally, a value must be set in the class
+    within which this variable is declared (i.e., intent='out').
+
+    Parameters
+    ----------
+    groups : str or list, optional
+        Variable group(s).
+    description : str, optional
+        Short description of the variable.
+    attrs : dict, optional
+        Dictionnary of additional metadata.
+
+    """
+    metadata = {
+        "var_type": VarType.OBJECT,
+        "intent": VarIntent.OUT,
+        "groups": _as_group_tuple(groups, None),
+        "attrs": attrs or {},
+        "description": description,
+    }
+
+    return attr.attrib(metadata=metadata, init=False, repr=False)
+
+
 def foreign(other_process_cls, var_name, intent="in"):
     """Create a reference to a variable that is defined in another
     process class.