Documentation - Developers Guide

[ccoffrin]

Add a special documentation guide / tutorial for developing new variable formulations and problem specifications.

[yeesian]

Relatedly, but separately (while writing the docs), I also have a running list of observations/FAQs that incoming developers might want to be aware of:

1. Implementation of expansion planning #33 (comment) Regarding constraint_complex_voltage,

   • semantics: "all of constraints that link your voltage variables together"
   • In the case of a non-convex model, where there is no variable lifting, this does nothing.
   • In the case of a relaxations, which have lifting, this links the voltage variables together.
   • It's not always specific to the lines, for example in the QC relaxation this is where the constraint w_i = v_i^2 would go. The SDP also has a similar property.

2. Implementation of expansion planning #33 (comment) The Matpower format was never intended to support EP. I am thinking that we should extend it (and the corresponding json format) with the fields we need for this application, so that you can specify for example which corridors can be built and which cannot.

3. Implementation of expansion planning #33 (comment) TNEP (Transmission Network Expansion) is currently considered as a specific optimization problem, and NE (network expansion) be more general functions that could be re-used in other problem definitions.

   • One could imagine a CE (capacity expansion) problem, and a problem that does both. To that end, all of the new variables related to NE have an _ne postfix. See also Cleaning up TNEP line on/off constraints #67.
   • It is worth to exploring how we could avoid repeating code across _ne line constraints and _on_off line constraints, however for now I suggest we leave that as a point in the issue tracker for later consideration.

4. Use "!" for fcns that modify inputs #24 Functions that modify their inputs typically have an exclamation point at the end of their names as a visual convention. But many users of this code may be non-julia experts, and having to put ! on everything will seem confusing / tedious to them. One of the design goals is to make PowerModels feel like JuMP. So the abstractions will seem familiar to users who know JuMP. Hence, we should follow the JuMP convention and not to use !.

5. Variable Name Conventions, and constructor return values (see Variable constructor return values #9 and Tests for Variable and Constraint return values #10).

6. Extensions to Matpower Format Parsing #53 (comment) The raw text of large JSON files is not easy for humans to read, comprehend, or edit. One needs a JSON viewer and/or a custom viewer, like the one I developed for the GRG JSON format, to understand the data. In contrast, it is easy for people to read and debug small-ish matrices in the Matpower format.

   • PowerModel's current JSON format is considered an internal thing, which can also be used for data exchange between algorithms.
   • If we are going to have a new network data standard, I would prefer something more wholistic, like GRG. For now, I see Matpower as the standard data exchange format for research, hence we should have good support for it. In time, I hope we can transition to something better, but until then, I think we should have good support for Matpower data.

Not sure if it'll be of interest to anyone to document/maintain such stuff.

[ccoffrin]

Very nice summary. This can be a starting point for developer documentation.

One clarification regarding points 2 and 3. Since v0.3, these issues around TNEP data formats have been addressed. The proposed format does not yet have documentation, but the tnep files in the test directory provide an example. The point about code duplication on _ne and _on_off is still open, but is not something I want to address until we have more experience supporting a wider range of problem formulations.

For context, GRG is a JSON-based network data format we are developing as part of the GRID DATA program. It should widely available in the next year or two. The long term goal is that PowerModels will be able to solve problems specified in GRG data.

[yeesian]

I don't yet understand why there is an AbstractPowerModel that isn't being used anywhere else in the codebase, when all the methods are only before defined for GenericPowerModel. What's the abstraction/interface for them?

[yeesian]

For context, GRG is a JSON-based network data format we are developing as part of the GRID DATA program. It should widely available in the next year or two.

Is the development/documentation for GRG meant to be kept internal/private during this period of time?

[ccoffrin]

There should be a alpha version of GRG released in the next few months. I will keep you posted.

GenericPowerModel{T} is a parametric concrete datatype which is the primary data store for PowerModels. The type parameter T is used to specialize the GenericPowerModel datatype, so that dynamic dispatch can be used to build the correct power model implementation.

AbastractPowerModel is the root of the type hierarchy of types that can be used to specialize GenericPowerModel. In theory, every PowerModels function using a GenericPowerModel should be defined as,

function pm_fun{T <: AbastractPowerModel}(pm::GenericPowerModel{T}; ...)
  ...
end

But for the most general functions I have often left the explicit typing off and used,

function pm_fun(pm::GenericPowerModel; ...)
  ...
end

If you think explicit typing of the most generic functions would make things easier to understand, we can surely add it.

[yeesian]

Based on the definition:

type GenericPowerModel{T<:AbstractPowerFormulation} <: AbstractPowerModel

It seems you're referring to AbstractPowerFormulation, rather than AbstractPowerModel?

[ccoffrin]

Right, that is correct. I think at this point there is no longer a need for an AbstractPowerModel.

[yeesian]

Right, that is correct. I think at this point there is no longer a need for an AbstractPowerModel.

For a while, I was under the impression AbstractPowerModel might be the root of the type hierarchy for the problem specification, and that there might be e.g.

PFModel <: AbstractPowerModel
OPFModel <: AbstractPowerModel
OTSModel <: AbstractPowerModel
TNEPModel <: AbstractPowerModel

and that the current variable/constraint methods defined for GenericPowerModel might (at some point) be defined for AbstractPowerModel, based on an interface that anything which is a subtype of AbstractPowerModel should implement.

Because otherwise, anything defined for GenericPowerModel{T} can be defined on T instead, making GenericPowerModel itself a redundant type.

Edit: I'm not advocating for it; just raising it to clarify why I'm asking these questions.

[ccoffrin]

Good to know this is confusing. In the short term I think we should remove AbstractPowerModel. In the longer term it is worth considering if a PowerModel types could simplify how one builds a problem formulation.

[yeesian]

Based on our discussion today, it seems like there might be a case for e.g.

abstract AbstractPowerModel
DeterministicPowerModel <: AbstractPowerModel
StochasticPowerModel <: AbstractPowerModel
...

instead of using GenericPowerModel for everything. Unless it's preferred for the stochasticity to be specified in the PowerFormulation?

[ccoffrin]

The GenericPowerModel hierarchy is used for encoding different forms of the same underlying problem. If your problem requires new data or new constraints, then it is a new problem class, which should be specified like the files appearing in the prob directory.