Add per tech cyclic storage (#543)

CHANGELOG.md

@@ -18,6 +18,8 @@ For instance, `flow_cap` can be defined per `carrier`.
 
 |new| Shadow prices obtained from a dual LP problem can be read by using `model.backend.shadow_prices.get("constraint_name")`
 
+|changed| |backwards-incompatible| `run.cyclic_storage` configuration option moved to being per-technology (`techs.tech_name.cyclic_storage: ...`).
+
 |changed| |backwards-incompatible| `file=/df=` parameter values as references to timeseries data is replaced with loading tabular data at the top-level using the `data_sources` key.
 
 |changed| Automatically derived transmission link distances default to kilometres, with the configuration option (`config.init.distance_unit`) to switch to the old default of distances in metres.

docs/advanced/constraints.md

@@ -170,12 +170,12 @@ In the latter case, Calliope will compute distances automatically based on the l
 
 For any technologies with storage (`storage` technologies or those with [storage buffer](#activating-storage-buffers-in-non-storage-technologies)), it is possible to link the storage at either end of the timeseries using the `cyclic_storage` parameter.
 This allows the user to better represent multiple years by just modelling one year.
-Cyclic storage is activated by default (to deactivate: `config.build.cyclic_storage: false`).
+Cyclic storage is activated by default (to deactivate: `cyclic_storage: false` in all storage technologies).
 As a result, a technology's initial stored energy at a given location will be equal to its stored energy at the end of the model's last timestep.
 
 For example, for a model running over a full year at hourly resolution, the initial storage at `Jan 1st 00:00` will be forced equal to the storage at the end of the timestep `Dec 31st 23:00`.
 By setting `storage_initial` for a technology, it is also possible to fix the value in the last timestep.
-For instance, with `config.build.cyclic_storage: true` and a `storage_initial` of zero, the stored energy *must* be zero by the end of the time horizon.
+For instance, with `cyclic_storage: true` and a `storage_initial: 0`, the stored energy *must* be zero by the end of the time horizon.
 
 Without cyclic storage in place, technologies can have any amount of stored energy by the end of the timeseries.
 This may prove useful in some cases, but makes little sense if you imagine your technologies operating in the same manner year-on-year.

docs/custom_math/syntax.md

@@ -35,7 +35,7 @@ Configuration options are any that are defined in `config.build`, where you can
 
     ??? example annotate "Examples"
 
-        - If you want to apply a constraint only if the configuration option `config.build.cyclic_storage` is _True_, you would include `config.cyclic_storage=True` (`True`/`False` is case insensitive).
+        - If you want to apply a constraint only if the configuration option `config.build.mode` is _operate_, you would include `config.mode=operate`.
         - If you want to apply a constraint across all `nodes` and `techs`, but only where the `flow_eff` parameter is less than 0.5, you would include `flow_eff<0.5`.
         - If you want to apply a constraint only for the first timestep in your timeseries, you would include `timesteps=get_val_at_index(dim=timesteps, idx=0)`. (1)
         - If you want to apply a constraint only for the last timestep in your timeseries, you would include `timesteps=get_val_at_index(dim=timesteps, idx=-1)`.
@@ -67,7 +67,7 @@ These statements will be combined first.
 
 ??? example "Examples"
 
-    - If you want to apply a constraint for `storage` technologies if the configuration option `cyclic_storage` is activated and it is the last timestep of the series: `base_tech=storage and config.cyclic_storage=True and timesteps=get_val_at_index(dim=timesteps, idx=-1)`.
+    - If you want to apply a constraint for `storage` technologies if the configuration option `cyclic_storage` is activated and it is the last timestep of the series: `base_tech=storage and cyclic_storage=True and timesteps=get_val_at_index(dim=timesteps, idx=-1)`.
     - If you want to create a decision variable for the input carriers of conversion technologies: `carrier_in and base_tech=conversion`
     - If you want to apply a constraint if the parameter `source_unit` is `energy_per_area` or the parameter `area_use_per_flow_cap` is defined: `source_unit=energy_per_area or area_use_per_flow_cap`.
     - If you want to apply a constraint if the parameter `flow_out_eff` is less than or equal to 0.5 and `source_use` has been defined, or `flow_out_eff` is greater than 0.9 and `source_use` has not been defined: `(flow_out_eff<=0.5 and source_use) or (flow_out_eff>0.9 and not source_use)`.

docs/migrating.md

@@ -515,6 +515,32 @@ Therefore, `24H` is equivalent to `24` in v0.6 if you are using hourly resolutio
 !!! warning
     Although we know that `operate` mode works on our example models, we have not introduced thorough tests for it yet - proceed with caution!
 
+### Per-technology cyclic storage
+
+The configuration option to set cyclic storage globally (`run.cyclic_storage`) has been moved to a parameter at the technology level.
+With this change, you can decide if a specific storage technology (or [technology with a storage buffer](#storage-buffers-in-all-technology-base-classes)) has cyclic storage enforced or not.
+As in v0.6, cyclic storage defaults to being _on_ (`cyclic_storage: true`).
+
+=== "v0.6"
+
+    ```yaml
+    run:
+      cyclic_storage: true
+    ```
+
+=== "v0.7"
+
+    ```yaml
+    techs:
+      storage_tech_with_cyclic_storage:
+        base_tech: storage
+        cyclic_storage: true
+      supply_tech_without_cyclic_storage:
+        base_tech: supply
+        include_storage: true
+        cyclic_storage: false
+    ```
+
 ## Removals
 
 ### `_equals` constraints
@@ -572,7 +598,7 @@ Instead, `supply_plus` can be effectively represented by using `supply` as the b
 To reimplement arbitrary links between carrier "tiers" (`in_2`, `out_2` etc.), you can define your own custom math, which is a simultaneously more powerful and more human-readable way of defining complex conversion technologies.
 
 !!! info "See also"
-    [Example of custom math to link carrier flows](examples/urban_scale/index.md#sparkles-interlude-custom-math).
+    [Example of custom math to link carrier flows](examples/urban_scale/index.md#interlude-custom-math).
 
 ### `carrier` key
 
@@ -670,8 +696,6 @@ Instead, data source filepaths should always be relative to the `model.yaml` fil
 Instead, it is possible to index any parameter over the time dimension.
 It is up to you to ensure the math formulation is set up to handle this change, which may require [tweaking existing math](custom_math/customise.md#introducing-custom-math-to-your-model).
 * With the [removal of time clustering](#clustering), we have removed `model.random_seed` and `model.time` options.
-* We have removed `model.objective`.
-With the introduction of our own math syntax, it is possible to add your own objective using [custom math](custom_math/index.md) and then activate/deactivate your objective of choice in the math.
 
 ### Plotting
 

src/calliope/config/config_schema.yaml

@@ -83,12 +83,6 @@ properties:
             type: string
             default: pyomo
             description: Module with which to build the optimisation problem
-          cyclic_storage:
-            type: boolean
-            default: true
-            description: >
-              If true, link storage levels in the last model timestep with the first model timestep.
-              `inter_cluster_storage` custom math must be included if using time clustering and setting this to `true`.
           ensure_feasibility:
             type: boolean
             default: false

src/calliope/config/model_data_checks.yaml

@@ -38,4 +38,8 @@ fail:
   - where: (storage_cap_max OR storage_cap_min) AND storage_cap_per_unit
     message: Cannot define both `storage_cap_per_unit` and `storage_cap_max`/`storage_cap_min`
 
-warn: []
+warn:
+  - where: cyclic_storage=True AND (base_tech=storage OR include_storage=True) AND config.mode=operate
+    message: >-
+      Cyclic storage should be switched off in operate mode.
+      Since the parameter defaults to True, make sure you explicitly set `cyclic_storage: false` for all your technologies with storage.

src/calliope/config/model_def_schema.yaml

@@ -329,7 +329,7 @@ properties:
 
           integer_dispatch:
             type: boolean
-            default: False
+            default: false
             x-resample_method: first
             x-type: float
             title: Integer dispatch switch.
@@ -578,7 +578,7 @@ properties:
             title: Storage capacity per purchased unit.
             description: >-
               Set the storage capacity of each integer unit of a technology purchased.
-              Unit: Unit: $\frac{\text{energy}}{\text{unit}}$.
+              Unit: $\frac{\text{energy}}{\text{unit}}$.
 
           storage_discharge_depth:
             $ref: "#/$defs/TechParamNullNumber"
@@ -607,7 +607,18 @@ properties:
             title: Storage loss rate
             description: >-
                Rate of storage loss per hour, used to calculate lost stored flow as `(1 - storage_loss)^hours_per_timestep`.
-               Unit: Unit: $\frac{\text{fraction}}{\text{hour}}$.
+               Unit: $\frac{\text{fraction}}{\text{hour}}$.
+
+          cyclic_storage:
+            type: boolean
+            default: true
+            x-type: float
+            title: Cyclic storage switch.
+            description: >
+              If true, link storage levels in the last model timestep with the first model timestep.
+              `inter_cluster_storage` custom math must be included if using time clustering and setting this to `true`.
+              This must be set to `false` if using `operate` mode.
+              Unit: boolean.
 
           purchased_units_min_systemwide:
             $ref: "#/$defs/TechParamNullNumber"

src/calliope/example_models/national_scale/scenarios.yaml

@@ -72,7 +72,6 @@ overrides:
         mode: operate
         operate_window: 12H
         operate_horizon: 24H
-        cyclic_storage: false
     nodes:
       region1.techs.ccgt.flow_cap: 30000
 
@@ -94,6 +93,8 @@ overrides:
       region1_to_region1_1.flow_cap: 9000
       region1_to_region1_2.active: false
       region1_to_region1_3.flow_cap: 2281
+      csp.cyclic_storage: false
+      battery.cyclic_storage: false
 
   check_feasibility:
     config:

src/calliope/math/base.yaml

@@ -189,17 +189,17 @@ constraints:
       - expression: storage == $storage_previous_step + source_use * source_eff - flow_out_inc_eff
     sub_expressions:
       storage_previous_step: &storage_previous_step
-        - where: timesteps=get_val_at_index(timesteps=0) AND NOT config.cyclic_storage=True
+        - where: timesteps=get_val_at_index(timesteps=0) AND NOT cyclic_storage=True
           expression: storage_initial * storage_cap
         - where: >-
             (
-              (timesteps=get_val_at_index(timesteps=0) AND config.cyclic_storage=True)
+              (timesteps=get_val_at_index(timesteps=0) AND cyclic_storage=True)
               OR NOT timesteps=get_val_at_index(timesteps=0)
             ) AND NOT lookup_cluster_first_timestep=True
           expression: (1 - storage_loss) ** roll(timestep_resolution, timesteps=1) * roll(storage, timesteps=1)
         - where: >-
             lookup_cluster_first_timestep=True AND NOT
-            (timesteps=get_val_at_index(timesteps=0) AND NOT config.cyclic_storage=True)
+            (timesteps=get_val_at_index(timesteps=0) AND NOT cyclic_storage=True)
           expression: >-
             (1 - storage_loss) **
             select_from_lookup_arrays(timestep_resolution, timesteps=lookup_cluster_last_timestep) *
@@ -255,7 +255,7 @@ constraints:
       Fix the relationship between carrier stored in a `storage` technology at
       the start and end of the whole model period.
     foreach: [nodes, techs]
-    where: "storage AND storage_initial AND config.cyclic_storage=True"
+    where: "storage AND storage_initial AND cyclic_storage=True"
     equations:
       - expression: >-
           storage[timesteps=$final_step] * (

src/calliope/math/storage_inter_cluster.yaml

@@ -9,15 +9,15 @@ constraints:
   balance_supply_with_storage:
     sub_expressions:
       storage_previous_step: &storage_previous_step
-        - where: timesteps=get_val_at_index(timesteps=0) AND NOT config.cyclic_storage=True
+        - where: timesteps=get_val_at_index(timesteps=0) AND NOT cyclic_storage=True
           expression: storage_initial * storage_cap
         - where: >-
             (
-              (timesteps=get_val_at_index(timesteps=0) AND config.cyclic_storage=True)
+              (timesteps=get_val_at_index(timesteps=0) AND cyclic_storage=True)
               OR NOT timesteps=get_val_at_index(timesteps=0)
             ) AND NOT lookup_cluster_last_timestep
           expression: (1 - storage_loss) ** roll(timestep_resolution, timesteps=1) * roll(storage, timesteps=1)
-        - where: lookup_cluster_last_timestep AND NOT (timesteps=get_val_at_index(timesteps=0) AND NOT config.cyclic_storage=True)
+        - where: lookup_cluster_last_timestep AND NOT (timesteps=get_val_at_index(timesteps=0) AND NOT cyclic_storage=True)
           expression: "0"
 
   balance_storage:
@@ -83,14 +83,14 @@ constraints:
       - expression: storage_inter_cluster == $storage_previous_step + $storage_intra
     sub_expressions:
       storage_previous_step:
-        - where: datesteps=get_val_at_index(datesteps=0) AND NOT config.cyclic_storage=True
+        - where: datesteps=get_val_at_index(datesteps=0) AND NOT cyclic_storage=True
           expression: storage_initial
-        - where: (datesteps=get_val_at_index(datesteps=0) AND config.cyclic_storage=True) OR NOT datesteps=get_val_at_index(datesteps=0)
+        - where: (datesteps=get_val_at_index(datesteps=0) AND cyclic_storage=True) OR NOT datesteps=get_val_at_index(datesteps=0)
           expression: ((1 - storage_loss) ** 24) * roll(storage_inter_cluster, datesteps=1)
       storage_intra:
-        - where: datesteps=get_val_at_index(datesteps=0) AND NOT config.cyclic_storage=True
+        - where: datesteps=get_val_at_index(datesteps=0) AND NOT cyclic_storage=True
           expression: "0"
-        - where: NOT (datesteps=get_val_at_index(datesteps=0) AND NOT config.cyclic_storage=True)
+        - where: NOT (datesteps=get_val_at_index(datesteps=0) AND NOT cyclic_storage=True)
           expression: storage[timesteps=$final_step]
     slices:
       final_step:

tests/common/lp_files/balance_storage.lp

@@ -0,0 +1,50 @@
+\* Source Pyomo model name=None *\
+
+min
+objectives(dummy_obj)(0):
++2.0 ONE_VAR_CONSTANT
+
+s.t.
+
+c_e_constraints(my_constraint)(b__test_storage__2005_01_01_00_00)_:
++1 variables(storage)(b__test_storage__2005_01_01_00_00)
++1.1111111111111112 variables(flow_out)(b__test_storage__electricity__2005_01_01_00_00)
+-1.0 variables(flow_in)(b__test_storage__electricity__2005_01_01_00_00)
+= 0.0
+
+c_e_constraints(my_constraint)(a__test_storage__2005_01_01_00_00)_:
++1 variables(storage)(a__test_storage__2005_01_01_00_00)
+-0.99 variables(storage)(a__test_storage__2005_01_01_01_00)
++1.1111111111111112 variables(flow_out)(a__test_storage__electricity__2005_01_01_00_00)
+-1.0 variables(flow_in)(a__test_storage__electricity__2005_01_01_00_00)
+= 0.0
+
+c_e_constraints(my_constraint)(a__test_storage__2005_01_01_01_00)_:
+-0.99 variables(storage)(a__test_storage__2005_01_01_00_00)
++1 variables(storage)(a__test_storage__2005_01_01_01_00)
++1.1111111111111112 variables(flow_out)(a__test_storage__electricity__2005_01_01_01_00)
+-1.0 variables(flow_in)(a__test_storage__electricity__2005_01_01_01_00)
+= 0.0
+
+c_e_constraints(my_constraint)(b__test_storage__2005_01_01_01_00)_:
+-0.99 variables(storage)(b__test_storage__2005_01_01_00_00)
++1 variables(storage)(b__test_storage__2005_01_01_01_00)
++1.1111111111111112 variables(flow_out)(b__test_storage__electricity__2005_01_01_01_00)
+-1.0 variables(flow_in)(b__test_storage__electricity__2005_01_01_01_00)
+= 0.0
+
+bounds
+   1 <= ONE_VAR_CONSTANT <= 1
+   0 <= variables(storage)(b__test_storage__2005_01_01_00_00) <= +inf
+   0 <= variables(flow_out)(b__test_storage__electricity__2005_01_01_00_00) <= +inf
+   0 <= variables(flow_in)(b__test_storage__electricity__2005_01_01_00_00) <= +inf
+   0 <= variables(storage)(a__test_storage__2005_01_01_00_00) <= +inf
+   0 <= variables(storage)(a__test_storage__2005_01_01_01_00) <= +inf
+   0 <= variables(flow_out)(a__test_storage__electricity__2005_01_01_00_00) <= +inf
+   0 <= variables(flow_in)(a__test_storage__electricity__2005_01_01_00_00) <= +inf
+   0 <= variables(flow_out)(a__test_storage__electricity__2005_01_01_01_00) <= +inf
+   0 <= variables(flow_in)(a__test_storage__electricity__2005_01_01_01_00) <= +inf
+   0 <= variables(storage)(b__test_storage__2005_01_01_01_00) <= +inf
+   0 <= variables(flow_out)(b__test_storage__electricity__2005_01_01_01_00) <= +inf
+   0 <= variables(flow_in)(b__test_storage__electricity__2005_01_01_01_00) <= +inf
+end

tests/common/test_model/energy_cap_per_storage_cap.yaml

@@ -6,7 +6,6 @@ config:
 
   build:
     mode: plan
-    cyclic_storage: False # necessary so demand can be fed from stored energy
     ensure_feasibility: true
   solve:
     solver: cbc
@@ -23,6 +22,7 @@ techs:
     base_tech: storage
     storage_initial: 1.0
     lifetime: 60
+    cyclic_storage: false # necessary so demand can be fed from stored energy
     cost_flow_cap:
       data: 1500000
       index: monetary

tests/test_math.py

@@ -135,6 +135,21 @@ def test_balance_transmission(self, compare_lps):
         }
         compare_lps(model, custom_math, "balance_transmission")
 
+    def test_balance_storage(self, compare_lps):
+        "Test balance storage with one tech having and one tech not having per-tech cyclic storage."
+        self.TEST_REGISTER.add("constraints.balance_storage")
+        model = build_test_model(
+            {
+                "nodes.a.techs.test_storage.cyclic_storage": True,
+                "nodes.b.techs.test_storage.cyclic_storage": False,
+            },
+            "simple_storage,two_hours",
+        )
+        custom_math = {
+            "constraints": {"my_constraint": model.math.constraints.balance_storage}
+        }
+        compare_lps(model, custom_math, "balance_storage")
+
     @pytest.mark.xfail(reason="not all base math is in the test config dict yet")
     def test_all_math_registered(self, base_math):
         "After running all the previous tests in the class, the base_math dict should be empty, i.e. all math has been tested"