Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Electricity infrastructure prototyping framework
Python JavaScript CSS
branch: master

Merge pull request #110 from SEL-Columbia/intersect_patch

This REALLY addresses the *settlement point ON network* problem
latest commit 478752fc0f
@chrisnatali chrisnatali authored
Failed to load latest commit information.
deployment prefix python with its dir for now (until we improve the way we manag…
docs added candidate segment intersection fix and minor mod to run_scenario
np.egg-info Added modKruskal test, full scenario run test (compare result to know…
np for real this time. This addresses the *settlement point ON network* …
test_data cleanup
utilities added candidate segment intersection fix and minor mod to run_scenario
.Rhistory Trying to solve myterious FuelCostPerKilowattHour reset of default va…
.gitignore cleanup
404.html modified NP dependencies for ubuntu and added nginx config, 404 and 5…
500.html modified NP dependencies for ubuntu and added nginx config, 404 and 5…
MANIFEST.in Fixed MANIFEST.in
README.rst forgot the unset note
TODO.txt Archived TODO items from invisibleroads.com for later use
default.cfg Made scenario.show lazy load node details to accommodate slower conne…
development.ini Moved private SVN repository to public Git repository
ez_setup.py Started with fresh template
migrate_jobs.sql updated sql migration for postgres
network_params.json for real this time. This addresses the *settlement point ON network* …
networkplanner.conf clean up
production.ini Moved private SVN repository to public Git repository
requirements.txt downgraded WebTest ver
requirements_network.txt Updated README with dependency & setup info and broke out requirement…
restart comment out data dir deletion since it appears that viewing scenarios…
sample_metric_params.json added configuration for tests
sample_metric_params_aliases.json added sample metric param input json files (for build_demand.py) and …
setup.cfg Fixed MANIFEST.in again
setup.py added candidate segment intersection fix and minor mod to run_scenario
test.ini Started with fresh template

README.rst

NP Electricity Infrastructure Prototyping Framework

NetworkPlanner is a framework for planning large-scale electricity infrastructure projects. Included are an example technology pricing metric model and a network optimization algorithm. The framework is extensible so that governments can adapt the example models to fit their country's needs. NetworkPlanner is developed and maintained by the Sustainable Engineering Lab at the Earth Institute of Columbia University.

Overview

The NetworkPlanner Web application exposes the concept of a Scenario as the main object by which users interact with the system. Users submit Scenarios with an associated set of inputs to be processed. The user can retrieve processed scenario data or view it via the web-site.

Dependencies

NetworkPlanner depends on the following system and python tools

  • System Tools
GDAL
Libspatialindex (only for network model)
PostgreSQL (only for production website)
RabbitMQ (only for distributed system)
  • Python Tools (not everything, see requirements.txt for all)
numpy/scipy
shapely

Setup

The recommended setup procedure for NetworkPlanner depends on the mode of operation.

Development and CLI

For Development and "CLI" (command-line) mode, the following script can be used as a guide to create a working environment on Debian based systems (including Ubuntu).

  • Sample bash script for basic Debian/Ubuntu setup
# setup system level packages
sudo apt-get -y install git python-pip python-dev libgdal-dev build-essential
# for scipy
sudo apt-get -y install gfortran libopenblas-dev liblapack-dev

# libspatialindex (only required for network model)
mkdir -p tmp
cd tmp
git clone git://github.com/libspatialindex/libspatialindex.git
sudo apt-get install -y automake libtool
(cd libspatialindex && ./autogen.sh && ./configure && make && sudo make install)
sudo ldconfig
cd

git clone https://github.com/SEL-Columbia/networkplanner.git np
# git clone git@github.com:SEL-Columbia/networkplanner.git np

# install virtualenv and virtualenvwrapper (see http://virtualenvwrapper.readthedocs.org)
# YMMV
# This is optional if you don't mind intermingled python packages on your system
sudo apt-get -y install python-virtualenv
sudo pip install virtualenvwrapper
# setup your bashrc to source virtualenvwrapper
if ! grep 'virtualenvwrapper\.sh' .bashrc;
then
cat >> .bashrc << EOF
. `which virtualenvwrapper.sh`
EOF
fi
. `which virtualenvwrapper.sh` # source it for the session (not needed for future sessions)
if ! workon np > /dev/null
then
    mkvirtualenv --system-site-packages np # from here on, working in np virtualenv
    cd np
    setvirtualenvproject # from here on you can enter 'workon np' to work in the np virtualenv
fi

# setup project specific python packages
pip install -r requirements.txt

# only required for network model
pip install -r requirements_network.txt

# NOTE:  Depending on your default user environment variables, you may need to set
# the LD_LIBRARY_PATH in order for networkplanner to reference libspatialindex properly
# Here are the contents of my $VIRTUAL_ENV/bin/postactivate file:

export OLD_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

# and the $VIRTUAL_ENV/bin/predeactivate file:

export LD_LIBRARY_PATH=$OLD_LD_LIBRARY_PATH
unset OLD_LD_LIBRARY_PATH

Modes of Operation

NetworkPlanner consists of 2 main modules (the Metric "builder" and the Network "builder"). These are exposed through both command-line interfaces and a web interface that can be deployed as a "Standalone" server or as a set of distributed processing servers.

CLI (aka command-line)

CLI mode allows a user to run the various utility scripts provided as part of NetworkPlanner without running a web/application server. This is useful for developing models and analyzing their output. These scripts reside in the utilities directory and can be run via python.

  • Sample Commands
# Run metric model on a set of demand nodes (using mvMax5 model)
# The output can be loaded as an R or Pandas dataframe for analysis
python utilities/build_demand.py mvMax5 sample_metric_params.json test_data/sample_demand_nodes.csv > sample_demand_out.csv

# Run full scenario (including network builder) on a set of demand nodes
# (the results will end up in the mv5_run directory as spec'd by the 5th param)
python utilities/run_scenario.py mvMax5 sample_metric_params.json modKruskal network_params.json mv5_run test_data/sample_demand_nodes.csv

# Create a dot graph for the dependencies of the MiniGrid RecurringCost
# variable of the mvMax4 model
# The dot file can be passed to graphviz utilities to render the graphs as png, svg, pdf...
python utilities/model_var_dependencies.py mvMax4 costMiniGrid.MiniGridSystemRecurringCostPerYear mv4_mg_rec.dot class

# Create a flat list of class, option/section, alias name mappings of all
# variables "under" Metric for mvMax5
python utilities/write_variable_fields.py mvMax5 Metric > mapping_mvmax5.csv

Development

Development mode runs the NetworkPlanner web-site via the Paste server with debugging enabled. This mode is useful for developing and testing the system and web interface. SQLite is the default database for this mode.

  • Some useful commands:
# Run the development server via paster
# 'ds' for development server
./restart ds

# Generate the docs
./restart docs

# start interactive python session for working with
# web app in development mode
paster shell development.ini

# Process scenarios submitted

Production (Standalone and Distributed)

The NetworkPlanner web application can be deployed as a standalone server or as a set of distributed processors:

  • Standalone: Single server handling all requests

  • Distributed:

    Master server handling main interface requests and then distributing the processing of scenarios among processor nodes. Utilizes RabbitMQ to synchronize between Master and Processors.

For Production deployment, we utilize a combination of Chef and Fabric to standardize and automate deployment. Production deployments utilize Postgresql as the main database.

Please refer to the networkplanner-devops repository for details and instructions.

Troubleshooting

  • Run the following script if RabbitMQ seems down
deployment/cluster-queue-reset.sh
Something went wrong with that request. Please try again.