New model definition YAML structure

[brynpickering]

This is something we've gone over a number of times in different issues (#362 , #483 , #452 , #324), so I thought I would move it to a discussion topic to consolidate different ideas and introduce new ones.

definitions as lists

This is something we were quite keen on in #362 , moving from:

techs:
  my_tech: ...

to:

techs:
  - id: my_tech
    ...

However, it makes overriding the model definition quite difficult. When doing the overriding, we have to search through the loaded lists for matching IDs and then apply the overrides to the dictionary contained in that list item. It is much more straightforward to use existing attrdict union functionality to achieve the same result, but with IDs as keys...

Perhaps if we move to loading the model definition in as pydantic objects, then lists of dicts would make more sense.

Top-level parameters

Introduced as an possibility in #452, I've since looked into it and had to change the syntax proposal. It is not possible to have tuples in the YAML and it is not possible to have a list as a key. So the proposed syntax changes from:

parameters:
  reserve_margin: {electricity: 0.1}
  bigM: 1e9
  objective_cost_class: {"costs": {"monetary": 1, "emissions": 0.1}}
  my_param: {("costs", "carriers"): {("monetary", "electricity"): 1, ("monetary", "heat"): 0.1}}
  spores_slack: 0.1
  spores_objective_cost_class: {"costs": {spores_score: 1, monetary: 0} }

to:

  bigM:
    data: 1e9
  objective_cost_class:
    data: 1  # will apply the same value to every index item
    index: [monetary, emissions]
    dims: costs
  my_custom_param:
    data: [0.6, 0.5]  # will apply a different value to each index item
    index: [[dual_fuel_in, monetary, biofuel], [dual_fuel_in, monetary, coal]]
    dims: [techs, costs, carriers]

It is also straightforward to add a new dimension to the model this way:

# Adds parameters for simple piecewise optimisation
parameters:
  cost_flow_cap_piecewise_slopes: 
    data: [5, 7, 14]
    index: [0, 1, 2]
    dims: pieces

  cost_flow_cap_piecewise_intercept:
    data: [0, -2, -16]
    index: [0, 1, 2]
    dims: pieces

Multiple carriers

As I have proposed in #483, carriers could be limited to _in/_out as single values or lists of values. The same syntax as for top-level parameters could be applied at the tech-level to handle carrier-specific parameter values:

techs:
  dual_fuel_in:
    parent: conversion
    carrier_in: [biofuel, coal]
    carrier_out: electricity
    flow_out_eff: 0.7
    flow_in_eff:
      data: [0.6, 0.5]
      index: [biofuel, coal]
      dims: [carriers]

This parameter structure could then be how the data is handled internally, converting the above into the below by prepending the tech name to index items and techs to dims:

parameters:
  parent:
    data: [conversion]
    index: [dual_fuel_in]
    dims: [techs]
  carrier_in:  # would need a trigger somewhere to tell calliope that this is a "lookup" array and so data should move to the index and the array should then become boolean.
    data: True
    index: [[dual_fuel_in, biofuel], [dual_fuel_in, coal]]
    dims: [techs, carriers]
  carrier_out:  # would need a trigger somewhere to tell calliope that this is a "lookup" array and so data should move to the index and the array should then become boolean.
    data: True
    index: [[dual_fuel_in, electricity]]
    dims: [techs, carriers]
  flow_out_eff:  # has gone from a single value to a parameter dictionary.
    data: 0.7
    index: [dual_fuel_in]
    dims: [techs]
  flow_in_eff:
    data: [0.6, 0.5]
    index: [[dual_fuel_in, biofuel], [dual_fuel_in, coal]]
    dims: [techs, carriers]

Costs

Costs always need to be linked to a cost class, and this could be done with the proposed parameter syntax, too:

techs:
  dual_fuel_in:
    cost_flow_out:
      data: 0.6
      index: monetary
      dims: costs

Or, if defining multiple carriers:

techs:
  dual_fuel_in:
    cost_flow_in:
      data: [0.6, 0.5]
      index: [[monetary, biofuel], [monetary, coal]]
      dims: [costs, carriers]

The YAML schema could enforce that cost parameters are dictionaries with the keys "data", "index" and "dims", and that "dims" must contain "costs".

Links

I still don't have a clear idea of how to proceed on this, since it is perhaps less about how the user defines links, but how they are then structured within the processed model data. At any rate, the parameter definition approach described here would also work within link_tech and link definition. Otherwise, I would refer to #324 for now.

[sjpfenninger]

Summary of discussion just now:

Definitions as lists

Stay with dicts of dicts, and leave pydantic aside for now.

Top-level parameters

Use the proposed syntax, but allow single-value definitions:

parameters:
  bigM: 1e9

This internally gets magically turned into:

parameters:
  bigM:
    data: 1e9

Multiple carriers / Costs

A separate top-level config to say which parameters are lookup tables and optionally what their dims are (in any case, these should never become Pyomo objects - so good to treat them separately).

Then processing any parameter would go through:

• Is it a lookup table? If yes, apply the lookup table magic - so carrier_in: [biofuel, coal] will still work
• Else: is it a single value? If yes, magic as described in Top-level parameters applies
• Else: it has to be dict with keys data, index, dims

Costs could be dealt with slightly different in the "make-it-a-dict-magic" case: a cost could get a default dims of costs and default index of monetary - that allows you to define a monetary cost simply as cost_flow_out: 10.

This could all be embedded in the schema, but the schema cannot be easily appended by users?

Links

Still needs more thought. One idea worth thinking more about is to use named links but track their flows separately, e.g. with an unconstrained flow variable or something like export and import.

[brynpickering]

A few thoughts on links:

Define in own top-level key

link_techs:
  foo:  # force explicit naming of a link 
    from: A
    to: B
    distance: 10
    one_way: false
    flow_in_eff: 0.95
    flow_cap_max: 100

Advantages:

• Clear delineation of these techs that act differently
• Can easily process it to apply the link tech to both nodes

Disadvantages:

• Requires explicit link naming
• How does inheriting of some base tech work? E.g., if everything is the same except from and to, where would one define that? tech_groups? techs?

Define link in top-level key, referencing transmission techs

links:
  foo:  # force explicit naming of a link 
    from: A
    to: B
    distance: 10
    one_way: false
    tech: ac_transmission
techs:
  ac_transmission:
    parent: transmission
    flow_in_eff: 0.95
    flow_cap_max: 100

Advantages:

• Can define technology characteristics and then associated them with links (less verbose, overall).
• Can easily process it to apply the link tech to both nodes
• Similar to current syntax except that links are only allowed to represent on technology. Other links for other technologies can exist between the same nodes, but will need a different link key.

Disadvantages:

• ac_transmission might cause confusion: does one add it to the tech list in a node or not? (I think we would say that it should never be allowed, it only acts as a grouping of data for links)

Define link in top-level key, referencing transmission techs in their own top-level-key

links:
  foo:  # force explicit naming of a link 
    from: A
    to: B
    distance: 10
    one_way: false
    tech: ac_transmission
link_techs:
  ac_transmission:
    parent: transmission
    flow_in_eff: 0.95
    flow_cap_max: 100

Advantages:

• Can define technology characteristics and then associated them with links (less verbose, overall).
• Can easily process it to apply the link tech to both nodes

Disadvantages:

• Do the number of our top-level keys start to get out of hand?

Define links at nodes

(could also be any of the other above ideas)

nodes:
  A:
    links:
      foo:  # force explicit naming of a link 
        to: B
        distance: 10
        one_way: false
        flow_in_eff: 0.95
        flow_cap_max: 100

Advantages:

• Techs and links for a node are all in one place.

Disadvantages:

• Not clear whether you have to also define it at B (I would say that you don't have to if we have a to key).
• What if they do define it at B too? Maybe even to a different to than A? Lots of sense checks needed.

Do we need to and from?

With explicit link names, we would just end up with foo being a tech available at both nodes; directionality is unimportant (at both nodes, flow_in = export, flow_out = import).

We could have something like connects: [A, B]/links: [A, B] and then we can have a constraint that says sum of exports == sum of imports (after accounting for efficiency losses). Like carrier_in and carrier_out this would be a "lookup" parameter that would assign boolean Trues to nodes "A" and "B" in the definition_matrix for that link.

This would allow a user to define the link matrix as a square, boolean array.

Only problem is unidirectional flows. If using e.g., connects: [A, B], how does one enforce unidirectionality? Don't know. But it is the only time when to and from become meaningful.

Could we have more than two links??

What if links: [A, B] was links: [A, B, C]? This would allow e.g. flows from B to go to A and C and then the sum of imports at A and C have to equal the exports from B. As with carrier-specific parameter definitions in techs, one could do some thing similar for links:

links:
  foo:  # force explicit naming of a link 
    connects: [A, B, C]
    distance: 
      data: [10, 20, 30] # <- Not sure how this works tbh
      index: [A, B, C]
      dims: [nodes]
    one_way: false
    flow_cap_max:
      data: [10, 20, 30] # <- would apply to `flow_out` at each link (i.e., imports)
      index: [A, B, C]
      dims: [nodes]

Potential problems:

• We couldn't set symmetric_transmission as a constraint since there are different max capacities; but then you no longer have line capacities for links, but something akin to substation capacities. Applying costs to the existence of the link is then a bit difficult (as the costs would be per "substation").
• This extends to distance: what does a distance value at node A mean?? It would need to be a distance to somewhere...

So in answer to my posed question: probably limit connects to equal exactly two nodes.

[brynpickering]

We have introduced our new YAML syntax in v0.7. Our pre-release documentation details all the decisions made w.r.t. a new YAML model definition structure.