NP Electricity Infrastructure Prototyping Framework
NetworkPlanner is a framework for planning large-scale electricity infrastructure projects. Included are an example technology pricing metric model and a network optimization algorithm. The framework is extensible so that governments can adapt the example models to fit their country's needs. NetworkPlanner is developed and maintained by the Sustainable Engineering Lab at the Earth Institute of Columbia University.
The NetworkPlanner Web application exposes the concept of a Scenario as the main object by which users interact with the system. Users submit Scenarios with an associated set of inputs to be processed. The user can retrieve processed scenario data or view it via the web-site.
NetworkPlanner depends on the following system and python tools
- System Tools
GDAL Libspatialindex (only for network model) PostgreSQL (only for production website) RabbitMQ (only for distributed system)
- Python Tools (not everything, see requirements.txt for all)
The recommended setup procedure for NetworkPlanner depends on the mode of operation.
Development and CLI
For Development and "CLI" (command-line) mode, the following script can be used as a guide to create a working environment on Debian based systems (including Ubuntu).
- Sample bash script for basic Debian/Ubuntu setup
# setup system level packages sudo apt-get -y install git python-pip python-dev libgdal-dev build-essential # for scipy sudo apt-get -y install gfortran libopenblas-dev liblapack-dev # libspatialindex (only required for network model) mkdir -p tmp cd tmp git clone git://github.com/libspatialindex/libspatialindex.git sudo apt-get install -y automake libtool (cd libspatialindex && ./autogen.sh && ./configure && make && sudo make install) sudo ldconfig cd git clone https://github.com/SEL-Columbia/networkplanner.git np # git clone email@example.com:SEL-Columbia/networkplanner.git np # install virtualenv and virtualenvwrapper (see http://virtualenvwrapper.readthedocs.org) # YMMV # This is optional if you don't mind intermingled python packages on your system sudo apt-get -y install python-virtualenv sudo pip install virtualenvwrapper # setup your bashrc to source virtualenvwrapper if ! grep 'virtualenvwrapper\.sh' .bashrc; then cat >> .bashrc << EOF . `which virtualenvwrapper.sh` EOF fi . `which virtualenvwrapper.sh` # source it for the session (not needed for future sessions) if ! workon np > /dev/null then mkvirtualenv --system-site-packages np # from here on, working in np virtualenv cd np setvirtualenvproject # from here on you can enter 'workon np' to work in the np virtualenv fi # setup project specific python packages pip install -r requirements.txt # only required for network model pip install -r requirements_network.txt # NOTE: Depending on your default user environment variables, you may need to set # the LD_LIBRARY_PATH in order for networkplanner to reference libspatialindex properly # Here are the contents of my $VIRTUAL_ENV/bin/postactivate file: export OLD_LD_LIBRARY_PATH=$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH # and the $VIRTUAL_ENV/bin/predeactivate file: export LD_LIBRARY_PATH=$OLD_LD_LIBRARY_PATH unset OLD_LD_LIBRARY_PATH
Modes of Operation
NetworkPlanner consists of 2 main modules (the Metric "builder" and the Network "builder"). These are exposed through both command-line interfaces and a web interface that can be deployed as a "Standalone" server or as a set of distributed processing servers.
CLI (aka command-line)
CLI mode allows a user to run the various utility scripts provided as part of NetworkPlanner without running a web/application server. This is useful for developing models and analyzing their output. These scripts reside in the utilities directory and can be run via python.
- Sample Commands
# Run metric model on a set of demand nodes (using mvMax5 model) # The output can be loaded as an R or Pandas dataframe for analysis python utilities/build_demand.py mvMax5 sample_metric_params.json test_data/sample_demand_nodes.csv > sample_demand_out.csv # Run full scenario (including network builder) on a set of demand nodes # (the results will end up in the mv5_run directory as spec'd by the 5th param) python utilities/run_scenario.py mvMax5 sample_metric_params.json modKruskal network_params.json mv5_run test_data/sample_demand_nodes.csv # Create a dot graph for the dependencies of the MiniGrid RecurringCost # variable of the mvMax4 model # The dot file can be passed to graphviz utilities to render the graphs as png, svg, pdf... python utilities/model_var_dependencies.py mvMax4 costMiniGrid.MiniGridSystemRecurringCostPerYear mv4_mg_rec.dot class # Create a flat list of class, option/section, alias name mappings of all # variables "under" Metric for mvMax5 python utilities/write_variable_fields.py mvMax5 Metric > mapping_mvmax5.csv
Development mode runs the NetworkPlanner web-site via the Paste server with debugging enabled. This mode is useful for developing and testing the system and web interface. SQLite is the default database for this mode.
- Some useful commands:
# Run the development server via paster # 'ds' for development server ./restart ds # Generate the docs ./restart docs # start interactive python session for working with # web app in development mode paster shell development.ini # Process scenarios submitted
Production (Standalone and Distributed)
The NetworkPlanner web application can be deployed as a standalone server or as a set of distributed processors:
Standalone: Single server handling all requests
Master server handling main interface requests and then distributing the processing of scenarios among processor nodes. Utilizes RabbitMQ to synchronize between Master and Processors.
Please refer to the networkplanner-devops repository for details and instructions.
- Run the following script if RabbitMQ seems down