Update and merge network module

[luseverin]

Hi,
I am working on updating the network module developped by @Evelyn-M in order to eventually be able to merge it into main. I think it could benefit some refactoring and restructuration. I have started to think about it but would like to get some feedback before I go further. Currently the state of the module is this:

Branch feature/networks
Description: First version of the module
Contains:

• nw_base.py
• • class Network: stores data (nodes and edges)
• nw_calcs.py
• • class GraphCalcs: computations on graphs
• • class Graph(GraphCalcs): convert network to graph via igraph library
• nw_preps.py
• • class NetworkPreprocess: Preprocessing steps on networks
• • class RoadPreprocess(NetworkPreprocess): preprocessing specific to roads
• • class PowerlinePreprocess(NetworkPreprocess): preprocessing specific to powerlines
• nw_utils.py
• • class PowerFunctionalData: handle data specific to powerlines

Branch feature/nw_minimal
Description: an attempt to simplify feature/networks
Contains:

• nw_base.py
• • class Network: stores data (nodes and edges)
• • class Graph: convert network to graph via igraph library
• nw_calcs.py
• nw_preps.py
• nw_utils.py
• • class PowerFunctionalData: handle data specific to powerlines

Issues:

• feature/networks depends on unsourced external libraries
• Tests are missing
• Some useful functionalities might be missing

Suggestions

• Add class/submodule to compute impacts on networks using climada core
• Add helper functions to process network exposures using external libraries such as OSMNx, NetworkX,...
• Add some plotting functions
• Add tests

I have tried to summarize the desired structure in a class diagram, where I tried to follow UML (see attached).

Questions:

• Not sure of what needs to be included in the module, and what should be left to the users. For instance, this is the case for helpers functions to process exposures to networks.
• Not sure what needs to be programmed as a class, and what could simply be a submodule with functions. This is the case with GraphCalcs which was a class in networks but has been change to independent functions in nw_minimal.

Looking forward to your suggestions!

[luseverin]

I have prepared additional sketches:

• User journey; the use of the module from a user's perspective.
  
  Legend: circle = external components, square = processes
• Data flow; flow of data betwen the different submodules and external components.
  
  Legend: circle = external components, squares separated with a line = classes and submodules within the network module, full squares = external libraries and packages.

@chahank @peanutfun and @emanuel-schmid, please let me know your thoughts on this.

[luseverin]

After discussion with @peanutfun, here are some suggestions for the update of the module:

• The Graph class can be merged into the Network class, as essentially Graph is just a wrapper of igraph.graph objects and does not really have a purpose on its own
• Graph-based functions can be put in the Network (or another class) only if it serves a specific purpose / make things easier e.g. avoid repeated back-and-forth passing of inputs and arguments across the different functions. I will reflect on this.
• It might make sense to have the dependency parameters kept as attributes (table) of the network class, as these are used by many functions.
• Regarding an "impact()" function, calculating the direct impacts of a hazard on the network, the following implementation was proposed:

def impact(hazard, impfs, agg_method, **kwargs) → Network
1. network.exposure() → exposure
2. impact(hazard, exposure, impfs) → impact
3. impact_to_network(self, impact, agg_method, **kwargs) → network_new
4. network_new.cascade()
5. Return network_new

• A new network object should be returned after impact() is called. Additionally, cascade() could be taken out of the impact() and be called separately by the user.
• We can allow for hazard sets with multiple events in the impact() provided that the user inputs a suitable agg_method to aggregate the impact matrix. This would mean that the cascade is only done on the aggregated impact.
• Helpers (e.g. functions to compute stats on the network) can be kept as free functions but still mapped as methods in the Network class to allow the user to call Network.stat().
• Network can be saved as zips of two feather or parquet for the edges and nodes geodataframes.
• Ideally, dependencies to other packages should be limited (e.g. networkX, OsmNx). However, use of OsmNx can be mentioned in tuto.

[peanutfun]

Thanks for making the effort of summarizing our discussion! 🙌

[Evelyn-M]

Thanks a lot @luseverin for this super neat overview & for making the effort of getting it into petals main. Thanks @peanutfun for your suggestions!

A couple of comments:

1. Merging Graph (& -based functions) --> Networks : Sure, this makes the module structure much more straight-forward. However, the intial suggestion by @emanuel-schmid was to split it into a data-holding class (Networks) and into a "processing" class (Graph). I'm not sure how important this is besides cleanliness principles and whether the trade-off to making the module structure makes this acceptable (I'd say so ;) ).

2. dependency parameters as attributes could make sense indeed. but can they be easily structured like a df or a dict in class attributes? i would not spend to much effort on this feature.

3. a network-impact() function should perhaps have 1 more substeps:

   def impact(hazard, impfs, agg_method, **kwargs) → Network
   1. network.exposure() → exposure
   2. impact(hazard, exposure, impfs) → impact
   3. impact_to_network(self, impact, agg_method, **kwargs) → network_new
   4. network_new.cascade()
   5. network --> impact_after_cascade
   6. Return impact_after_cascade, network_new
   

4. Agree on the dependencies to networkX, OsmNx - if useful for you, do so, but don't bother with writing convenience functions for an unknown user group :)