Add method to load data sources into the model. (#532)

* Add `data-sources` top-level key to load tabular data into the model from file or pandas.DataFrame.
* Remove `file/df=` functionality.
* Fix resampler; add parameter dtype casting.
* parent -> base_tech.
* Add data source schema validation.
* Add tutorial to docs.

---------

Co-authored-by: Stefan Pfenninger <stefan@pfenninger.org>

CHANGELOG.md

@@ -4,6 +4,8 @@ v0.7 includes a major change to how Calliope internally operates. Most of this a
 
 ### User-facing changes
 
+|new| Any tabular data, not only timeseries data, can be loaded from file / an in-memory pandas DataFrame under the `data_sources` key.
+
 |new| Files containing user-defined mathematical formulations can be referenced in the model configuration. User-defined mathematical formulations must follow the new Calliope YAML math syntax (see "Internal changes" below).
 
 |new| Nodes can inherit from a newly added `node_groups` top-level key by referencing a node group with `inherit: ...` in the node configuration.
@@ -14,22 +16,24 @@ This enables un-indexed parameters to be defined, as well as those indexed over
 |new| parameter dimensions at the `tech` or `node` level can be enhanced using the new `parameter` definition syntax.
 For instance, `flow_cap` can be defined per `carrier`.
 
+|changed| |backwards-incompatible| `file=/df=` parameter values as references to timeseries data is replaced with loading tabular data at the top-level using the `data_sources` key.
+
 |changed| Automatically derived transmission link distances default to kilometres, with the configuration option (`config.init.distance_unit`) to switch to the old default of distances in metres.
 
 |changed| |backwards-incompatible| Costs must be defined using the new `parameter` definition syntax.
 
 |changed| `flow_cap` (formerly `energy_cap`) is indexed over `carriers` as well as `nodes` and `techs`.
 This allows capacities to be defined separately for input and output flows for `conversion` technologies.
 
-|changed| |backwards-incompatible| `_plus` technology groups have been removed.
-Now, `supply_plus` can be effectively represented by using `supply` as the technology parent and setting `include_storage: true` in the model definition.
-`conversion_plus` can be represented by using `conversion` as the technology parent and using lists of carriers in `carrier_in` and/or `carrier_out`.
+|changed| |backwards-incompatible| `_plus` base technologies have been removed.
+Now, `supply_plus` can be effectively represented by using `supply` as the technology base tech and setting `include_storage: true` in the model definition.
+`conversion_plus` can be represented by using `conversion` as the technology base tech and using lists of carriers in `carrier_in` and/or `carrier_out`.
 To represent `in_2`, `out_2` etc. carrier "tiers", you will need to define your own custom math.
 
-|changed| |backwards-incompatible| Technology inheritance has moved to the `inherit` key, leaving `parent` as a protected parameters that can only accept one of the abstract base technologies.
+|changed| |backwards-incompatible| Technology inheritance has moved to the `inherit` key, leaving `base_tech` to replace `parent` as a protected parameter that can only accept one of the abstract base technologies.
 
 |changed| |backwards-incompatible| Links are defined within `techs` and not in their own `links` section.
-Any technology with a `transmission` parent will require `to` and `from` nodes to be defined.
+Any technology with a `transmission` base technology will require `to` and `from` nodes to be defined.
 Then, all transmission links will be given their name as defined in `techs` rather than having the name be automatically derived by Calliope.
 
 |changed| |backwards-incompatible| Flow efficiencies are now split into inflow (`flow_in_eff`) and outflow (`flow_out_eff`) efficiencies. This enables different storage charge/discharge efficiencies to be applied.
@@ -51,7 +55,7 @@ Then, all transmission links will be given their name as defined in `techs` rath
 
 |changed| |backwards-incompatible| `get_formatted_array` has been removed in favour of directly accessing the data, since we no longer concatenate sets. E.g., `model.get_formatted_array("storage")` -> `model.results.storage`.
 
-|changed| |backwards-incompatible| The dimensions of the model data no longer include all possible subsets. E.g. a user can no longer access `loc_techs_supply` to view the location/technology pairs which have defined `supply` as their top-level parent. Instead, the same subset can be supplied by calling `model.inputs.inheritance.str.endswith('supply')` to create a boolean array of technologies with `supply` as their top-level parent.
+|changed| |backwards-incompatible| The dimensions of the model data no longer include all possible subsets. E.g. a user can no longer access `loc_techs_supply` to view the location/technology pairs which have defined `supply` as their top-level base tech. Instead, the same subset can be supplied by calling `model.inputs.base_tech == "supply"` to create a boolean array of technologies with `supply` as their top-level base tech.
 
 |changed| |backwards-incompatible| Group constraints have been removed. They will be replaced by `custom constraint` functionality.
 

docs/css/extra.css

@@ -57,6 +57,11 @@
     font-size: 0.8em;
 }
 
+/* reduce font size of code blocks in notebook markdown cells */
+.jp-MarkdownOutput div.highlight {
+    font-size: 0.8em;
+}
+
 /* Allow tables to be horizontally scrollable. */
 .md-typeset__scrollwrap {
   overflow-x: auto;

docs/custom_math/examples/annual_energy_balance.yaml

@@ -69,7 +69,7 @@ constraints:
       Limit total flow out of the system into a sink that is not pinned
       by `sink_use_equals`.
     foreach: [techs]
-    where: parent=demand AND annual_sink_max
+    where: base_tech=demand AND annual_sink_max
     equations:
       - expression: >
           sum(flow_in, over=[nodes, carriers, timesteps]) <= annual_sink_max

docs/custom_math/examples/max_time_varying.yaml

@@ -4,7 +4,7 @@
 #     Set per-timestep variations in limits to out/inflows, which would otherwise be limited by a static value.
 #     For example, `flow_cap` can be made to fluctuate per timestep above/below its rated value.
 #     User-defined timeseries parameters need to be in the model inputs for these constraints.
-#     This can be achieved by defining them for each relevant technology in your YAML and linking it to a CSV file (e.g., `techs.my_tech.constraints.flow_cap_max_relative_per_ts: file=my_file.csv`).
+#     This can be achieved by defining them for each relevant technology in a CSV file and loading that as a [data source][data-sources].
 #
 #     New top-level parameters:
 #

docs/custom_math/examples/net_import_share.yaml

@@ -54,6 +54,6 @@ constraints:
 global_expressions:
   flow_out_transmission_techs:
     foreach: [nodes, techs, carriers, timesteps]
-    where: parent=transmission
+    where: base_tech=transmission
     equations:
       - expression: flow_out

docs/custom_math/syntax.md

@@ -42,13 +42,13 @@ Configuration options are any that are defined in `config.build`, where you can
 
     1.  `get_val_at_index` is a [helper function][helper-functions]; read more below!
 
-1. Checking the `parent` of a technology (`storage`, `supply`, etc.) or its inheritance chain (if using `tech_groups` and the `inherit` parameter).
+1. Checking the `base_tech` of a technology (`storage`, `supply`, etc.) or its inheritance chain (if using `tech_groups` and the `inherit` parameter).
 
     ??? examples
 
-        - If you want to create a decision variable across only `storage` technologies, you would include `parent=storage`.
+        - If you want to create a decision variable across only `storage` technologies, you would include `base_tech=storage`.
         - If you want to apply a constraint across only your own `rooftop_supply` technologies (e.g., you have defined `rooftop_supply` in `tech_groups` and your technologies `pv` and `solar_thermal` define `#!yaml inherit: rooftop_supply`), you would include `inheritance(rooftop_supply)`.
-        Note that `parent=...` is a simple check for the given value of `parent`, while `inheritance()` is a helper function ([see below][helper-functions]) which can deal with the fact that intermediate groups may be present, e.g. `pv` might inherit from `rooftop_supply` which in turn might inherit from `electricity_supply`.
+        Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a helper function ([see below][helper-functions]) which can deal with the fact that intermediate groups may be present, e.g. `pv` might inherit from `rooftop_supply` which in turn might inherit from `electricity_supply`.
 
 1. Subsetting a set.
 The sets available to subset are always [`nodes`, `techs`, `carriers`] + any additional sets defined by you in [`foreach`][foreach-lists].
@@ -67,8 +67,8 @@ These statements will be combined first.
 
 ??? examples
 
-    - If you want to apply a constraint for `storage` technologies if the configuration option `cyclic_storage` is activated and it is the last timestep of the series: `parent=storage and config.cyclic_storage=True and timesteps=get_val_at_index(dim=timesteps, idx=-1)`.
-    - If you want to create a decision variable for the input carriers of conversion technologies: `carrier_in and parent=conversion`
+    - If you want to apply a constraint for `storage` technologies if the configuration option `cyclic_storage` is activated and it is the last timestep of the series: `base_tech=storage and config.cyclic_storage=True and timesteps=get_val_at_index(dim=timesteps, idx=-1)`.
+    - If you want to create a decision variable for the input carriers of conversion technologies: `carrier_in and base_tech=conversion`
     - If you want to apply a constraint if the parameter `source_unit` is `energy_per_area` or the parameter `area_use_per_flow_cap` is defined: `source_unit=energy_per_area or area_use_per_flow_cap`.
     - If you want to apply a constraint if the parameter `flow_out_eff` is less than or equal to 0.5 and `source_use` has been defined, or `flow_out_eff` is greater than 0.9 and `source_use` has not been defined: `(flow_out_eff<=0.5 and source_use) or (flow_out_eff>0.9 and not source_use)`.
 

docs/examples/logging.py → docs/examples/calliope_logging.py

@@ -6,11 +6,11 @@
 #       extension: .py
 #       format_name: percent
 #       format_version: '1.3'
-#       jupytext_version: 1.11.2
+#       jupytext_version: 1.16.0
 #   kernelspec:
-#     display_name: Python 3
+#     display_name: calliope_docs_build
 #     language: python
-#     name: python3
+#     name: calliope_docs_build
 # ---
 
 # %% [markdown]
@@ -35,7 +35,7 @@
 
 # %%
 calliope.set_log_verbosity("info")
-m = calliope.examples.national_scale()
+m = calliope.examples.national_scale(time_subset=["2005-01-01", "2005-01-01"])
 m.build()
 m.solve()
 
@@ -44,7 +44,7 @@
 
 # %%
 calliope.set_log_verbosity("info", include_solver_output=False)
-m = calliope.examples.national_scale()
+m = calliope.examples.national_scale(time_subset=["2005-01-01", "2005-01-01"])
 m.build()
 m.solve()
 
@@ -53,7 +53,7 @@
 
 # %%
 calliope.set_log_verbosity("debug")
-m = calliope.examples.national_scale()
+m = calliope.examples.national_scale(time_subset=["2005-01-01", "2005-01-01"])
 m.build()
 m.solve()
 
@@ -82,7 +82,7 @@
 
 # You can also use logging in your scripts to add more information
 logger.info("Loading the national-scale example model")
-m = calliope.examples.national_scale()
+m = calliope.examples.national_scale(time_subset=["2005-01-01", "2005-01-01"])
 
 logger.info("Building the national-scale example model optimisation problem")
 m.build()
@@ -119,7 +119,7 @@
 
 # You can also use logging in your scripts to add more information
 logger.info("Loading the national-scale example model")
-m = calliope.examples.national_scale()
+m = calliope.examples.national_scale(time_subset=["2005-01-01", "2005-01-01"])
 
 logger.info("Building the national-scale example model optimisation problem")
 m.build()
@@ -159,7 +159,7 @@
 
 # You can also use logging in your scripts to add more information
 logger.info("Loading the national-scale example model")
-m = calliope.examples.national_scale()
+m = calliope.examples.national_scale(time_subset=["2005-01-01", "2005-01-01"])
 
 logger.info("Building the national-scale example model optimisation problem")
 m.build()
@@ -197,6 +197,6 @@
     file_handler.setFormatter(formatter)
     logging.getLogger(logger_name).addHandler(file_handler)
 
-m = calliope.examples.national_scale()
+m = calliope.examples.national_scale(time_subset=["2005-01-01", "2005-01-01"])
 m.build()
 m.solve()

docs/examples/calliope_model_object.py

@@ -6,11 +6,11 @@
 #       extension: .py
 #       format_name: percent
 #       format_version: '1.3'
-#       jupytext_version: 1.11.2
+#       jupytext_version: 1.16.1
 #   kernelspec:
-#     display_name: calliope_dev
+#     display_name: calliope_docs_build
 #     language: python
-#     name: python3
+#     name: calliope_docs_build
 # ---
 
 # %% [markdown]