Introduce nomopyomo

[FabianHofmann]

Introduce optimization without usage of pyomo, based on nomopyomo which is way faster and memory efficient for especially large networks.

What happened so far:

New Modules

• linopf.py - includes all code for setting up the problem
• linopt.py - includes all solver (io to solver) relevant code
• stats.py - include code for post-optimization analysis

Ordinary features of lopf without pyomo:

• include ramp limits
• include Storage Units
• calculate nodal imbalances
• constant term in objective function
• calculate voltage angles
• implement glpk solver
• logfile for cbc

Additional features

• global constraints for transmission expansion volume/cost limit
• ability to fix values via '_set' for arbitrary variables, also static variables (so f.e. n.generators.p_nom_set can fix values for extendable p_nom variables)
• warm start
• add iterative lopf (ilopf), updating impedances after each solving process
• docstrings for linop{t,f}.py
• add docstrings to documentation (+ incorporate changes)
• ensure windows compatibility
• add windows and osx in travis CI

Other changes

• determine_network_topology now includes find_slackbus and find_cycles

• add new lopf to tests in

  1. acdc test
  2. storage unit test

• skip python2.7 test in travis file. I had to much python3 code syntax, and didn't see the nessecity to rewrite it as so many package give up on python2 support. But perhaps you have a different view.

The network.lopf function got a new argument as proposed in #96, so it becomes possible now to call

network.lopf(pyomo=False)

[fneum]

This already looks super nice! I just added "docstrings in linop{t,f}.py and documentation" to the remaining todos.

[fneum]

I'm fine with dropping Python 2.7 support in future versions. This discussion already came up in #79 . Obviously, we should make this clear in release notes and documentation. Additionally, we should keep a documentation version of the last Python 2.7 support version of PyPSA in https://pypsa.readthedocs.io.

[FabianHofmann]

This already looks super nice! I just added "docstrings in linop{t,f}.py and documentation" to the remaining todos.

Thanks! And yes. Could you help me there? Perhaps with the documention part, always missing the patience to do it properly. Considering the docstrings, most of the function in linopt already have it already.
Missing parts are

• define_ functions in linopf.py (which probably should just get a small note)
• prepare_lopf and assign_solution in linopf.py
• run_and_read_ functions in linopt.py

[fneum]

Could you help me there?

Sure, I can do that. WIP will be here: https://pypsa.readthedocs.io/en/nomopyomo/

[nworbmot]

Great! I've just integrated the code into the solve_network of PyPSA-Eur-Sec. It mostly works, but pne thing that breaks is the test of status and termination_condition because pyomo returns ok for status on success whereas nomopyomo returns optimal. Could you make this similar to pyomo?

[FabianHofmann]

done :)

[nworbmot]

Brilliant, thank you! Just running a test for it now.

[nworbmot]

Total memory usage (PyPSA+gurobi) is 10% lower with new integer storing of variable and constraint references! Good result!

[nworbmot]

I seem to be having the problem that the results are not saved in the netCDF correctly - could it be that the reading out of results with the new integer scheme is broken?

[nworbmot]

BTW My changes to the PyPSA-Eur-Sec solving script (very similar to PyPSA-Eur) you can find here:
https://github.com/nworbmot/pypsa-eur-sec/tree/nomopyomo

[FabianHofmann]

I seem to be having the problem that the results are not saved in the netCDF correctly - could it be that the reading out of results with the new integer scheme is broken?

I'm wondering, could you try the following snippet:

import pypsa
import os

network_path = os.path.join(pypsa.__path__[0], '..', 'examples', 'ac-dc-meshed',
                            'ac-dc-data')
n = pypsa.Network(network_path)
n.lopf(pyomo=False)

n.export_to_netcdf('test.nc')
n2 = pypsa.Network('test.nc')
print(n.generators_t.p)
print(n2.generators_t.p)

It works fine for me. The only thing that happens is the order of the generators which is sorted alphabetically afterwards and was aligned to n.generators.index before.

[nworbmot]

Hey, I've solved the problem with the above commit. The problem was with gurobi in linopt - the termination_condition was being set to status, i.e. "ok", which is then rejected as "Problem was not solved to optimality" in linopf.
BTW Here's the new memory picture - consumption is less than a third of Pyomo!

[FabianHofmann]

Hey, I've solved the problem with the above commit. The problem was with gurobi in linopt - the termination_condition was being set to status, i.e. "ok", which is then rejected as "Problem was not solved to optimality" in linopf.

aah, sorry about that, overlooked it

[nworbmot]

No worries. One thing we may have to deal with later, is that some large problems don't solve with "ok", but with "warning", which is why @coroa wrote this exception:
https://github.com/PyPSA/pypsa-eur/blob/master/scripts/solve_network.py#L294
The results are still useable, so we may need a work around. I suggest to fix this when it arises, since I can't remember exactly what the status should be...

[nworbmot]

Hey, I'm just finalising the implementation of nomopyomo in PyPSA-Eur. I'm having a problem when adding a line volume constraint to the problem which already has a CO2 constraint. E.g. take the AC-DC example from PyPSA/examples and add before LOPF:

n.links["carrier"] = "DC"

n.add("GlobalConstraint",
          "line_volume_constraint",
          type="transmission_volume_expansion_limit",
          carrier_attribute="AC,DC",
          sense="<=",
          constant=1e6)

This results in an error:

--> 617     define_global_constraints(n, snapshots)
~/energy/playground/nomopyomo/pypsa/pypsa/linopf.py in define_global_constraints(n, sns)
    505         rhs = glc.constant
    506         con = write_constraint(n, lhs, sense, rhs, axes=pd.Index([name]))
--> 507         set_conref(n, con, 'GlobalConstraint', 'mu', name)
~/energy/playground/nomopyomo/pypsa/pypsa/linopt.py in set_conref(n, constraints, c, attr, spec)
    427         else:
    428             n.constraints.loc[idx[c, attr], :] = [pnl, spec]
--> 429         _add_reference(n.cons[c], constraints, attr, pnl=pnl)
~/energy/playground/nomopyomo/pypsa/pypsa/linopt.py in _add_reference(ref_dict, df, attr, pnl)
    385             ref_dict.df[attr] = df
    386         else:
--> 387             ref_dict.df.loc[df.index, attr] = df
KeyError: "None of [Index(['line_volume_constraint'], dtype='object')] are in the [index]"

Any ideas?

[FabianHofmann]

Could you try setting

ref_dict.df = pd.concat([ref_dict.df, df.to_frame(attr)])

instead where the error happens (387 in linopt)? I think it worked for me

[nworbmot]

That works for the AC-DC example! I'll try it on PyPSA-Eur now.

[nworbmot]

There was no error building problem for PyPSA-Eur now. Just waiting for it to solve. Did the error make sense, i.e. did you understand why it was going wrong?

[FabianHofmann]

There was no error building problem for PyPSA-Eur now. Just waiting for it to solve. Did the error make sense, i.e. did you understand why it was going wrong?

I'm not very sure why specifically this works now. Before it had to do with changing adding multiple values at the same time, like

df = pd.DataFrame(1, index=['a'], columns=[0])
df.loc[['b']] = 2

which breaks (I did not know this before). Now, what happens is rewriting the whole dataframe which is more stable.

[nworbmot]

PyPSA-Eur-Sec is now all working for arbitrary line volumes:
https://github.com/nworbmot/pypsa-eur-sec/tree/nomopyomo
I think we can aim for a new PyPSA release with it before Xmas? Anything else that needs sorting before we release?
Thank you @FabianHofmann for all your hard working integrating it!

[FabianHofmann]

@nworbmot this sounds good! Perhaps to little things:

1. Did you test a unit commitment model? the implementation allows only a light version of UC but it should work, a warning is given which parameters are ignored.

2. There is the new module stats.py which is supposed to enable double-checking the constraints post-optimization. Should we include it? Or any changes to made here? We could also merge it later...

[nworbmot]

Thanks @FabianHofmann for all your hard work on this! By far the biggest pull request we've ever had!