Network Wrangler 2.0

[sdrewc]

This commit implements several changes to NetworkWrangler described below.

1. Types
Types now include 'project', 'plan', and 'seed'.
1.a. Projects. These are the code representations of changes to a hwy or trn network. They are similar to projects in older versions of wrangler, with updates described below.
1.b. Plans. Plans are a collection of projects and/or other plans. They are specified by a file called planSpecs.csv with two columns: the first indicates the path/name of the project, and the second indicates the type.
1.c. Seed. These form the baseline starting point for a network. They are similar to projects but are meant to be applied first.

2. Project Upgrades.
NetworkWrangler expects now expects every 'project', 'plan', and 'seed' to have an __init__.py file with functions to return standard information. These are similar to __init__.py files used by previous versions of wrangler with changes described below.
2.a. champVersion. This now returns a tuple of the earliest and latest versions of CHAMP this project is compatible with. This is no longer stored in champVersion.txt for hwy projects.
2.b. wranglerVersion. This is a new function that returns a tuple of the earliest and latest versions of NetworkWrangler the a project is compatible with
2.c. desc. Returns a description of the project. This is no longer stored in desc.txt for hwy projects.
2.d. year. Returns the opening year of a project, or the model year if the modelyear keyword argument is supplied.
2.e. networks. This is a new function that returns a list of the network types that the project applies to. This is used by plans so projects listed in the plan are added to the correct networks.
2.f. prereqs. This returns a list of projects which must be applied before this project.
2.g. coreqs. This returns a list of projects which should be applied with this project (order doesn't matter though).
2.h. conflicts. This returns a list of projects which are known to cause errors when applied with this project.
2.i. apply, applyFiles, and applyEdits. These are used the same as before, but we recommend using with a modelyear argument to specify the opening year of a project.

3. Changes to build_network.py.
3.a. For each 'plan' in each network's project list, the script adds its projects, seeds, and other plans. These are all checked out prior to beginning to build any network.
3.b. For each 'project' in each network's project list (including projects contained within plans), the script checks the project list for prereqs, coreqs, and conflicts, and prompts the user for review and approval before continuing.
3.c. All projects (including projects contained within plans) are checked out prior to building any network. Only after checking out all projects and checking wranglerVersion, champVersion, prereqs, coreqs, and conflicts are networks built.

[e-lo]

I know you didn't write this @sdrewc , but eventually it would be good to create some tests with example data...

[e-lo]

I would look thru these and eliminate what we don't actually use in wrangler (maybe all?)

[sdrewc]

@e-lo I'll review them. At least some are necessary; I added them when I started running into missing dependencies, and they were all from this set.

[e-lo]

@sdrewc , This is a ton of work - thanks!

I may just not be aware of something else you were trying to accomplish with this PR, but I think there is potentially a lot of unused code that duplicates things in CHAMP (potentially detrimentally).

My personal opinion is that we should remove duplications that aren't being used so that they don't fall out of sync. i.e. if we change the ordering of disaggregate records in CHAMP but accidentally use the ordering in Wrangler....then we could be using completely wrong columns which could have bad outcomes.

I've noted a few instances of this in-line, but there might be others.

Something that might be helpful: http://www.pylint.org/

...and then again, I may be completely off. Let me know! I'll wait on the PR merge for now...

[sdrewc]

@e-lo,

So wrangler's main functions are to build networks, and to be used for transit assignment, right? I only changed the parts for the network building function. In the spirit of keeping wrangler whole and stand-alone I didn't touch any of the other stuff.

Do you prefer to split into two pieces, with this just network building parts in this project? I think that is a pretty big project in itself because there are so many interdependencies here.

Or maybe I'm misunderstanding. Should I pull out dataTable, Lookups, hdf5, etc? and just point to CHAMP? Then wrangler doesn't stand alone anymore, but maybe that's fine.

Drew

[e-lo]

@sdrewc - this is an interesting issue that I think warrants some discussion. @lmz might have some opinions on this as well.

The idea was to be able to have network wrangler work on its own, but to be able to "drop"
a static version of it into sf-champ and have it work there too.

There is a potential conflict in that we don't want to have to go to the "network wrangler" project to be able to edit the lookups that affect all of SF-CHAMP. And...we don't want there to end up being two lookups when you copy wrangler into SF-CHAMP. But...we want NetworkWrangler to be able to use the lookups. Basically we have a dependency circle of mutually assured destruction.

Here are some thoughts about how we could address this issue:

1. Have an _includes or _static folder in Wrangler that has all the things that it needs from SF-CHAMP in order to run by itself that would not be copied into SF-CHAMP.
2. The assignment portion just simply doesn't function w/out SF-CHAMP.

...but for files that are not ever used by wrangler, I think those are for sure out (i.e. disaggregate record?? maybe it is used but I can't think about how that would work right now...)

Thoughts?

[sdrewc]

@e-lo and @lmz I like the _static idea. So much, in fact, that I updated my pull request to include it. I removed unused dependencies: dataTable, Cube, and odict remain.