Update math docs and handling of missing data (#563)

* Add parameters to math documentation.
* Add cross-refs, defaults and unit data into math documentation for variables, global expressions, and parameters.
* Update handling of empty variable/expression array elements by using default values to fill NaNs.
* Move guide to math documentation to be in the preamble of each pre-defined math doc page. 
* Remove `noqa F811` with typing overloads now that Ruff understands them.

CHANGELOG.md

@@ -1,3 +1,19 @@
+## 0.7.0.dev3 (dev)
+
+### User-facing changes
+
+|new| List of pre-defined parameters given in the `pre-defined` math documentation, with references back to the constraints/variables/global expressions in which they are defined (either in the `expression` string or the `where` string).
+
+|new| Units and default values for variables and global expressions added to the math documentation.
+
+|new| Variables and global expressions can have a `default` value, which is used to fill missing array elements when doing math operations.
+These default values ensure that `NaN` doesn't creep into the built optimisation problem math and are set to values that lead to them having no impact on the optimal solution.
+
+|fixed| Timeseries clustering file can be a non-ISO standard date format.
+Both the index and the values of the timeseries (both being date strings) should be in the user-defined `config.init.time_format`.
+
+|changed| `inbuilt` math -> `pre-defined` math and `custom` math -> `pre-defined` math in the documentation.
+
 ## 0.7.0.dev2 (2024-01-26)
 
 v0.7 includes a major change to how Calliope internally operates.

docs/advanced/backend.md

@@ -12,7 +12,7 @@ You can use this interface to:
 The optimisation problem has input parameters, decision variables, global expressions, constraints, and an objective.
 The data of all these components are stored as [xarray.DataArray][]s and you can query the backend to inspect them.
 For instance, [`#!python model.backend.parameters`][calliope.backend.backend_model.BackendModel.parameters] will provide you with an [xarray.Dataset][] of input parameters transformed into mutable objects that are used in the optimisation.
-In addition to the input data you provided, these arrays fill in missing data with default values if the parameter is one of those [predefined in Calliope][model-definition-schema].
+In addition to the input data you provided, these arrays fill in missing data with default values if the parameter is one of those [predefined in Calliope][model-definition-schema] (the `Parameters` section of our [pre-defined base math documentation][base-math] shows where these parameters are used within math expressions).
 
 1. Update a parameter value.
 If you are interested in updating a few values in the model, you can run [`#!python model.backend.update_parameter`][calliope.backend.backend_model.BackendModel.update_parameter].

docs/advanced/constraints.md

@@ -3,7 +3,7 @@
 On this page, we look at some of the more advanced features of Calliope's math and configuration that can help you understand the breadth of what we offer.
 
 !!! info "See also"
-    [Base math formulation][base-math],
+    [Pre-defined math formulation][base-math] (which includes a description of our pre-defined parameters),
     [Model definition schema][model-definition-schema],
     [Introducing your own math to your model](../user_defined_math/customise.md),
     ["MILP" example model](../examples/milp/index.md).
@@ -88,7 +88,7 @@ techs:
     waste_per_flow_out: 0.1  # (1)!
 ```
 
-1. This is a user-defined parameter that you won't find in our [list of internal parameters][model-definition-schema].
+1. This is a user-defined parameter that you won't find in the `Parameters` section of our [pre-defined base math documentation][base-math].
 You can use it in your own math to link nuclear waste outflow with electricity outflow.
 
 ## Activating storage buffers in non-storage technologies

docs/hooks/generate_math_docs.py

@@ -5,13 +5,16 @@
 """
 
 import importlib.resources
+import logging
 import tempfile
 import textwrap
 from pathlib import Path
 
 import calliope
 from mkdocs.structure.files import File
 
+logger = logging.getLogger("mkdocs")
+
 TEMPDIR = tempfile.TemporaryDirectory()
 
 MODEL_PATH = Path(__file__).parent / "dummy_model" / "model.yaml"
@@ -20,6 +23,15 @@
 # {title}
 {description}
 
+## A guide to math documentation
+
+If a math component's initial conditions are met (the first `if` statement), it will be applied to a model.
+For each [objective](#objective), [constraint](#subject-to) and [global expression](#where), a number of sub-conditions then apply (the subsequent, indented `if` statements) to decide on the specific expression to apply at a given iteration of the component dimensions.
+
+In the expressions, terms in **bold** font are [decision variables](#decision-variables) and terms in *italic* font are [parameters](#parameters).
+The [decision variables](#decision-variables) and [parameters](#parameters) are listed at the end of the page; they also refer back to the global expressions / constraints in which they are used.
+Those parameters which are defined over time (`timesteps`) in the expressions can be defined by a user as a single, time invariant value, or as a timeseries that is [loaded from file or dataframe](../creating/data_sources.md).
+
 [:fontawesome-solid-download: Download the {math_type} formulation as a YAML file]({filepath})
 """
 
@@ -74,7 +86,7 @@ def write_file(
 
     files.append(
         File(
-            path=output_file,
+            path=output_file.as_posix(),
             src_dir=TEMPDIR.name,
             dest_dir=config["site_dir"],
             use_directory_urls=config["use_directory_urls"],
@@ -84,8 +96,8 @@ def write_file(
     # Append the source file to make it available for direct download
     files.append(
         File(
-            path=Path("math") / filename,
-            src_dir=importlib.resources.files("calliope"),
+            path=(Path("math") / filename).as_posix(),
+            src_dir=Path(importlib.resources.files("calliope")).as_posix(),
             dest_dir=config["site_dir"],
             use_directory_urls=config["use_directory_urls"],
         )
@@ -164,11 +176,21 @@ def _keep_only_changes(base_model: calliope.Model, model: calliope.Model) -> Non
                     full_del.append(name)
             else:
                 _add_to_description(component_dict, "|NEW|")
+
     model.math_documentation.build()
     for key in expr_del:
         model.math_documentation._instance._dataset[key].attrs["math_string"] = ""
     for key in full_del:
         del model.math_documentation._instance._dataset[key]
+    for var in model.math_documentation._instance._dataset.values():
+        var.attrs["references"] = var.attrs["references"].intersection(
+            model.math_documentation._instance._dataset.keys()
+        )
+        var.attrs["references"] = var.attrs["references"].difference(expr_del)
+
+    logger.info(
+        model.math_documentation._instance._dataset["carrier_in"].attrs["references"]
+    )
 
 
 def _add_to_description(component_dict: dict, update_string: str) -> None:

docs/migrating.md

@@ -329,7 +329,8 @@ Here are the main changes to parameter/decision variable names that are not link
 * `exists` → `active`.
 
 !!! info "See also"
-    [Our full list of internally defined parameters][model-definition-schema].
+    [Our full list of internally defined parameters][model-definition-schema];
+    the `Parameters` section of our [pre-defined base math documentation][base-math].
 
 ### Renaming / moving configuration options
 

docs/pre_defined_math/index.md

@@ -27,13 +27,3 @@ If you want to introduce new constraints, decision variables, or objectives, you
 See the [user-defined math](../user_defined_math/index.md) section for an in-depth guide to applying your own math.
 
 The pre-defined math can be explored in this section by selecting one of the options in the left-hand-side table of contents.
-
-## A guide to math documentation
-
-If a math component's initial conditions are met (those to the left of the curly brace), it will be applied to a model.
-For each objective, constraint and global expression, a number of sub-conditions then apply (those to the right of the curly brace) to decide on the specific expression to apply at a given iteration of the component dimensions.
-
-In the expressions, terms in **bold** font are decision variables and terms in *italic* font are parameters.
-A list of the decision variables is given at the end of this page.
-A detailed listing of parameters along with their units and default values is given in the [model definition reference sheet][model-definition-schema].
-Those parameters which are defined over time (`timesteps`) in the expressions can be defined by a user as a single, time invariant value, or as a timeseries that is [loaded from file or dataframe](../creating/data_sources.md).

docs/user_defined_math/components.md

@@ -35,6 +35,8 @@ Without a `where` string, all valid members (according to the `definition_matrix
     1. a reference to an input parameter, where each valid member of the variable (i.e. each value of the variable for a specific combination of indices) will get a different value based on the values of the referenced parameters (see example above).
     If a value for a valid variable member is undefined in the referenced parameter, the decision variable will be unbounded for this member.
 1. It can be deactivated so that it does not appear in the built optimisation problem by setting `active: false`.
+1. It can take on a `default` value that will be used in math operations to avoid `NaN` values creeping in.
+The default value should be set such that it has no impact on the optimisation problem if it is included (most of the time, this means setting it to zero).
 
 ## Global Expressions
 
@@ -60,6 +62,8 @@ Without a `where` string, all valid members (according to the `definition_matrix
 1. It has [equations](syntax.md#equations) (and, optionally, [sub-expressions](syntax.md#sub-expressions) and [slices](syntax.md#slices)) with corresponding lists of `where`+`expression` dictionaries.
 The equation expressions do _not_ have comparison operators; those are reserved for [constraints](#constraints)
 1. It can be deactivated so that it does not appear in the built optimisation problem by setting `active: false`.
+1. It can take on a `default` value that will be used in math operations to avoid `NaN` values creeping in.
+The default value should be set such that it has no impact on the optimisation problem if it is included (most of the time, this means setting it to zero).
 
 ## Constraints
 

docs/user_defined_math/syntax.md

@@ -212,7 +212,7 @@ Usually you shouldn't need to use this, as your `where` string will mask those N
 But if you're having trouble setting up your math, it is a useful function to getting it over the line.
 
 !!! note
-    Our [internally defined parameters][model-definition-schema] all have default values which propagate to the math.
+    Our internally defined parameters, listed in the `Parameters` section of our [pre-defined base math documentation][base-math] all have default values which propagate to the math.
     You only need to use `default_if_empty` for decision variables and global expressions, and for user-defined parameters.
 
 ## equations
@@ -329,4 +329,11 @@ equations:
 slices:
   tech_ref:
     - expression: lookup_techs
-```
+```
+
+## default
+
+Variables and global expressions can take `default` values.
+These values will be used to fill empty array elements (i.e., those that are not captured in the [`where` string](#where-strings)) when conducting math operations.
+A default value is not _required_, but is a useful way to ensure you do not accidentally find yourself with empty array elements creeping into the constraints.
+These manifest as `NaN` values in the optimisation problem, which will cause an error when the problem is sent to the solver.

src/calliope/backend/backend_model.py

@@ -56,7 +56,10 @@
 
 class BackendModelGenerator(ABC):
     _VALID_COMPONENTS: tuple[_COMPONENTS_T, ...] = typing.get_args(_COMPONENTS_T)
-    _COMPONENT_ATTR_METADATA = ["description", "unit"]
+    _COMPONENT_ATTR_METADATA = ["description", "unit", "default"]
+
+    _PARAM_DESCRIPTIONS = extract_from_schema(MODEL_SCHEMA, "description")
+    _PARAM_UNITS = extract_from_schema(MODEL_SCHEMA, "x-unit")
 
     def __init__(self, inputs: xr.Dataset, **kwargs):
         """Abstract base class to build a representation of the optimisation problem.
@@ -189,6 +192,7 @@ def _check_inputs(self):
             "input_data": self.inputs,
             "helper_functions": helper_functions._registry["where"],
             "apply_where": True,
+            "references": set(),
         }
         for check_type, check_list in check_results.items():
             for check in data_checks[check_type]:
@@ -290,7 +294,10 @@ def _add_component(
         )
 
         top_level_where = parsed_component.generate_top_level_where_array(
-            self, align_to_foreach_sets=False, break_early=break_early
+            self,
+            align_to_foreach_sets=False,
+            break_early=break_early,
+            references=references,
         )
         if break_early and not top_level_where.any():
             return parsed_component
@@ -309,7 +316,9 @@ def _add_component(
                 .astype(np.dtype("O"))
             )
         for element in equations:
-            where = element.evaluate_where(self, initial_where=top_level_where)
+            where = element.evaluate_where(
+                self, initial_where=top_level_where, references=references
+            )
             if break_early and not where.any():
                 continue
 
@@ -372,7 +381,6 @@ def _add_all_inputs_as_parameters(self) -> None:
             model_data (xr.Dataset): Input model data.
             defaults (dict): Parameter defaults.
         """
-
         for param_name, param_data in self.inputs.filter_by_attrs(
             is_result=0
         ).data_vars.items():
@@ -391,7 +399,7 @@ def _add_all_inputs_as_parameters(self) -> None:
                 "parameters", param_name, "Component not defined; using default value."
             )
             self.add_parameter(
-                param_name, xr.DataArray(default_val), use_inf_as_na=False
+                param_name, xr.DataArray(default_val), default_val, use_inf_as_na=False
             )
             self.parameters[param_name].attrs["is_result"] = 0
         LOGGER.info("Optimisation Model | parameters | Generated.")

src/calliope/backend/expression_parser.py

@@ -84,6 +84,7 @@ class EvalString(ABC):
     def __eq__(self, other):
         return self.__repr__() == other
 
+    @abstractmethod
     def __repr__(self) -> str:
         "Return string representation of the parsed grammar"
 
@@ -94,7 +95,7 @@ def as_math_string(self) -> str:
         """Return evaluated expression as a LaTex math string"""
 
     @abstractmethod
-    def as_array(self) -> xr.DataArray | list[str, float]:
+    def as_array(self) -> xr.DataArray | list[str | float]:
         """Return evaluated expression as an array"""
 
     @overload
@@ -104,7 +105,7 @@ def eval(self, return_type: Literal["math_string"], **eval_kwargs) -> str:
     @overload
     def eval(
         self, return_type: Literal["array"], **eval_kwargs
-    ) -> xr.DataArray | list[str, float]:
+    ) -> xr.DataArray | list[str | float]:
         """arrays evaluate to arrays"""
 
     def eval(
@@ -126,13 +127,13 @@ def eval(
             helper_functions (dict[str, type[ParsingHelperFunction]]): Dictionary of allowed helper functions.
             as_values (bool, optional): Return array as numeric values, not backend objects. Defaults to False.
         Returns:
-            Union[str, list[str, float], xr.DataArray]:
+            Union[str, list[str | float], xr.DataArray]:
                 If `math_string` is desired, returns a valid LaTex math string.
                 If `array` is desired, returns xarray DataArray or a list of strings/numbers (if the expression represents a list).
         """
 
         self.eval_attrs = eval_kwargs
-        evaluated: Union[str, list[str, float], xr.DataArray]
+        evaluated: Union[str, list[str | float], xr.DataArray]
         if return_type == "array":
             evaluated = self.as_array()
         elif return_type == "math_string":
@@ -357,6 +358,7 @@ def _eval(
 
     def _compare_bitwise(self, where: bool, lhs: Any, rhs: Any) -> Any:
         "Comparison function for application to individual elements of the array"
+
         if not where or pd.isnull(lhs) or pd.isnull(rhs):
             return np.nan
         match self.op:
@@ -407,12 +409,12 @@ def _arg_eval(self, return_type: Literal["math_string"], arg: Any) -> str:
     @overload
     def _arg_eval(
         self, return_type: Literal["array"], arg: Any
-    ) -> xr.DataArray | list[str, float]:
+    ) -> xr.DataArray | list[str | float]:
         "array return"
 
     def _arg_eval(
         self, return_type: RETURN_T, arg: Any
-    ) -> Union[str, xr.DataArray, list[str, float]]:
+    ) -> Union[str, xr.DataArray, list[str | float]]:
         "Evaluate the arguments of the helper function"
         if isinstance(arg, pp.ParseResults):
             evaluated = arg[0].eval(return_type, **self.eval_attrs)
@@ -600,12 +602,12 @@ def _eval(self, return_type: Literal["math_string"], as_values: bool) -> str:
     @overload
     def _eval(
         self, return_type: Literal["array"], as_values: bool
-    ) -> xr.DataArray | list[str, float]:
+    ) -> xr.DataArray | list[str | float]:
         "array return"
 
     def _eval(
         self, return_type: RETURN_T, as_values: bool
-    ) -> Union[str, xr.DataArray, list[str, float]]:
+    ) -> Union[str, xr.DataArray, list[str | float]]:
         "Evaluate the referenced `slice`."
         self.eval_attrs["as_values"] = as_values
         return self.eval_attrs["slice_dict"][self.name][0].eval(
@@ -615,7 +617,7 @@ def _eval(
     def as_math_string(self) -> str:
         return self._eval("math_string", False)
 
-    def as_array(self) -> xr.DataArray | list[str, float]:
+    def as_array(self) -> xr.DataArray | list[str | float]:
         evaluated = self._eval("array", True)
         if isinstance(evaluated, xr.DataArray) and evaluated.isnull().any():
             evaluated = evaluated.notnull()
@@ -705,7 +707,7 @@ def as_math_string(self) -> str:
         input_list = self.as_array()
         return "[" + ",".join(str(i) for i in input_list) + "]"
 
-    def as_array(self) -> list[str, float]:
+    def as_array(self) -> list[str | float]:
         values = [val.eval("array", **self.eval_attrs) for val in self.val]
         # strings and numbers are returned as xarray arrays of size 1,
         # so we extract those values.
@@ -732,6 +734,7 @@ def __repr__(self) -> str:
     def as_math_string(self) -> str:
         self.eval_attrs["as_values"] = False
         evaluated = self.as_array()
+        self.eval_attrs["references"].add(self.name)
 
         if evaluated.shape:
             dims = rf"_\text{{{','.join(str(i).removesuffix('s') for i in evaluated.dims)}}}"
@@ -753,6 +756,8 @@ def as_array(self) -> xr.DataArray:
         else:
             try:
                 evaluated = backend_interface._dataset[self.name]
+                if "default" in evaluated.attrs:
+                    evaluated = evaluated.fillna(evaluated.attrs["default"])
             except KeyError:
                 evaluated = xr.DataArray(self.name, attrs={"obj_type": "string"})