Add documentation for the math syntax (#456)

* Re-organise docs nav; move generation scripts to hooks
* Add base and custom math docs
* Clean up docstrings and api config
* Update reference page names

---------

Co-authored-by: Stefan Pfenninger <stefan@pfenninger.org>

.readthedocs.yml

@@ -6,8 +6,6 @@ build:
   tools:
     python: mambaforge-4.10
   jobs:
-    pre_build:
-      - python ./docs/pre_build/generate_math.py
     post_create_environment:
       - mamba install python=3.11 --file requirements/docs.txt
       - pip install --no-deps .

doc/conf.py

@@ -59,7 +59,8 @@
 ]
 
 # The suffix of source filenames.
-source_suffix = ".rst"
+source_suffix = {".rst": "restructuredtext", ".md": "markdown"}
+
 
 # A string of reStructuredText that will be included at the beginning of every
 # source file that is read

doc/index.rst

@@ -48,6 +48,7 @@ User guide
    user/building
    user/running
    user/analysing
+   user/math_def
    user/tutorials
    user/advanced_constraints
    user/advanced_features

docs/custom_math/components.md

@@ -0,0 +1,107 @@
+
+# Math components
+
+Here, we will briefly introduce each of the math components you will need to build an optimisation problem.
+A more detailed description of the math YAML syntax is provided on the [math syntax][math-syntax] page and in the [math formulation schema][math-formulation-schema].
+
+## Decision variables
+
+Decision variables (called `variables` in Calliope) are the unknown quantities whose values can be chosen by the optimisation algorithm while optimising for the chosen objective (e.g. cost minimisation) under the bounds set by the constraints.
+These include the output capacity of technologies, the per-timestep flow of carriers into and out of technologies or along transmission lines, and storage content in each timestep.
+A decision variable in Calliope math looks like this:
+
+```yaml
+variables:
+--8<-- "src/calliope/math/base.yaml:variable"
+```
+
+1. It needs a unique name (`storage_cap` in the example above).
+1. Ideally, it has a long-form `description` and a `unit` added.
+These are not required, but are useful metadata for later reference.
+1. It can have a top-level `foreach` list and `where` string.
+Without a `foreach`, it becomes an un-indexed variable.
+Without a `where` string, all valid members (according to the `definition_matrix`) based on `foreach` will be included in this decision variable.
+1. It can define a domain to turn it into a binary or integer variable (in either of those cases, domain becomes `integer`).
+1. It requires a minimum and maximum bound, which can be:
+    1. a numeric value:
+    ```yaml
+    variables:
+      flow_out:
+      ...
+        bounds:
+          min: 0
+          max: .inf
+    ```
+    1. a reference to an input parameter, where each valid member of the variable (i.e. each value of the variable for a specific combination of indices) will get a different value based on the values of the referenced parameters (see example above).
+    If a value for a valid variable member is undefined in the referenced parameter, the decision variable will be unbounded for this member.
+1. It can be deactivated so that it does not appear in the built optimisation problem by setting `active: false`.
+
+## Global Expressions
+
+Global expressions are those combinations of decision variables and input parameters that you want access to in multiple constraints / objectives in the model.
+You will also receive the result of the global expression as a numeric value in your optimisation results, without having to do any additional post-processing.
+
+For instance, total costs are global expressions as the cost associated with a technology is not a _constraint_, but rather a linear combination of decision variables and parameters (e.g., `storage_cap * cost_storage_cap`).
+To not clutter the objective function with all combinations of variables and parameters, we define a separate global expression:
+
+```yaml
+global_expressions:
+--8<-- "src/calliope/math/base.yaml:expression"
+```
+
+Global expressions are by no means necessary to include, but can make more complex linear expressions easier to keep track of and can reduce post-processing requirements.
+
+1. It needs a unique name (`cost` in the above example).
+1. Ideally, it has a long-form `description` and a `unit` added.
+These are not required, but are useful metadata for later reference.
+1. It can have a top-level `foreach` list and `where` string.
+Without a `foreach`, it becomes an un-indexed expression.
+Without a `where` string, all valid members (according to the `definition_matrix`) based on `foreach` will be included in this expression.
+1. It has [equations][] (and, optionally, [sub-expressions][] and [slices][]) with corresponding lists of `where`+`expression` dictionaries.
+The equation expressions do _not_ have comparison operators; those are reserved for [constraints][]
+1. It can be deactivated so that it does not appear in the built optimisation problem by setting `active: false`.
+
+## Constraints
+
+[Decision variables][decision-variables] / [global expressions][global-expressions] need to be constrained or included in the model objective.
+Constraining these math components is where you introduce the realities of the system you are modelling.
+This includes limits on things like the maximum area use of tech (there's only so much rooftop available for roof-mounted solar PV), and links between in/outflows such as how much carrier is consumed by a technology to produce each unit of output carrier.
+Here is an example:
+
+```yaml
+constraints:
+--8<-- "src/calliope/math/base.yaml:constraint"
+```
+
+1. It needs a unique name (`set_storage_initial` in the above example).
+1. Ideally, it has a long-form `description` and a `unit` added.
+These are not required, but are useful metadata for later reference.
+1. It can have a top-level `foreach` list and `where` string.
+Without a `foreach`, it becomes an un-indexed constraint.
+Without a `where` string, all valid members (according to the `definition_matrix`) based on `foreach` will be included in this constraint.
+1. It has [equations][] (and, optionally, [sub-expressions][] and [slices][]) with corresponding lists of `where`+`expression` dictionaries.
+The equation expressions _must_ have comparison operators.
+1. It can be deactivated so that it does not appear in the built optimisation problem by setting `active: false`.
+
+## Objectives
+
+With your constrained decision variables and a global expression that binds these variables to costs, you need an objective to minimise/maximise. The default built-in objective is `min_cost_optimisation` and looks as follows:
+
+```yaml
+objectives:
+--8<-- "src/calliope/math/base.yaml:objective"
+```
+
+1. It needs a unique name.
+1. Ideally, it has a long-form `description` and a `unit` added.
+These are not required, but are useful metadata for later reference.
+1. It can have a top-level `where` string, but no `foreach` (it is a single value you need to minimise/maximise).
+Without a `where` string, the objective will be activated.
+1. It has [equations][] (and, optionally, [sub-expressions][] and [slices][]) with corresponding lists of `where`+`expression` dictionaries.
+These expressions do _not_ have comparison operators.
+1. It can be deactivated so that it does not appear in the built optimisation problem by setting `active: false`.
+
+!!! warning
+
+    You can only have one objective activated in your math.
+    If you have loaded multiple, you can deactivate unwanted ones using `active: false`, or you can set your top-level `where` string on each that leads to only one being valid for your particular problem.

docs/custom_math/customise.md

@@ -0,0 +1,47 @@
+# Introducing custom math to your model
+
+Once you understand the [math components][math-components] and the [formulation syntax][math-syntax], you'll be ready to introduce custom math to your model.
+
+You can find examples of custom math that we have put together in the [custom math example gallery][custom-math-example-gallery].
+
+Whenever you introduce your own math, it will be applied on top of the [base math][base-math].
+Therefore, you can include base math overrides as well as add new math.
+For example, you may want to introduce a timeseries parameter to the built-in `storage_max` constraint to limit maximum storage capacity on a per-timestep basis:
+
+```yaml
+storage_max:
+  equations:
+    - expression: storage <= storage_cap * time_varying_parameter
+```
+
+The other elements of the `storage_max` constraints have not changed (`foreach`, `where`, ...), so we do not need to define them again when overriding the custom math.
+
+When defining your model, you can reference any number of YAML files containing the custom math you want to add in `config.init`. The paths are relative to your main model configuration file:
+
+```yaml
+config:
+  init:
+    custom_math: [my_new_math_1.yaml, my_new_math_2.yaml]
+```
+
+You can also define a mixture of your custom math and the [inbuilt math][inbuilt-math]:
+
+```yaml
+config:
+  init:
+    custom_math: [my_new_math_1.yaml, storage_inter_cluster, my_new_math_2.md]
+```
+
+## Writing your own math documentation
+
+You can write your model's mathematical formulation to view it in a rich-text format (as we do for our [inbuilt math][inbuilt-math] in this documentation).
+To write a LaTeX, reStructuredText, or Markdown file that includes only the math valid for your model:
+
+```python
+model = calliope.Model("path/to/model.yaml")
+model.build_math_documentation(include="valid")
+model.write_math_documentation(filename="path/to/output/file.[tex|rst|md]")
+```
+
+You can then convert this to a PDF or HTML page using your renderer of choice.
+We recommend you only use HTML as the equations can become too long for a PDF page.

docs/custom_math/annual_energy_balance.yaml → docs/custom_math/examples/annual_energy_balance.yaml

@@ -5,18 +5,18 @@
 #
 #   New technology-level parameters:
 #
-#   * `annual_flow_max`
-#   * `annual_source_max`
+#   - `annual_flow_max`
+#   - `annual_source_max`
 #
 #   New top-level parameters:
 #
-#   * `annual_flow_max` (if summing over technologies and/or nodes)
-#   * `flow_max_group` (if summing over technologies and/or nodes)
+#   - `annual_flow_max` (if summing over technologies and/or nodes)
+#   - `flow_max_group` (if summing over technologies and/or nodes)
 #
 #   Helper functions used:
 #
-#   * `inheritance` (where)
-#   * `sum` (expression)
+#   - `inheritance` (where)
+#   - `sum` (expression)
 #
 # ---
 
@@ -49,7 +49,7 @@ constraints:
     description: >
       Limit total flow into the system from a particular source.
       NOTE: this only works for supply_plus technologies.
-        For `supply` technologies you will need to convert `flow_out` to `flow_in` using `energy_eff` and limit that.
+      For `supply` technologies you will need to convert `flow_out` to `flow_in` using `energy_eff` and limit that.
     foreach: [techs]
     where: source_use AND annual_source_max
     equations: