Make link 'oneway' attribute optional when writing networks to disk

[mfitz]

See https://arupdigital.atlassian.net/browse/LAB-1002 and #55

Before this change

$ git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

$ python scripts/simplify_network.py -n /Users/mickyfitz/workspace/matesto/data/matsim12-test-town/network.xml -s /Users/mickyfitz/workspace/matesto/data/matsim12-test-town/transitschedule.xml -p epsg:27700 -od /tmp/genet

2021-02-16 21:32:27,877 - Reading in network at /Users/mickyfitz/workspace/matesto/data/matsim12-test-town/network.xml
2021-02-16 21:32:27,880 - Reading in schedule at /Users/mickyfitz/workspace/matesto/data/matsim12-test-town/transitschedule.xml
2021-02-16 21:32:28,306 - Simplifying the Network.
2021-02-16 21:32:28,306 - Begin simplifying the graph
2021-02-16 21:32:28,307 - Generating paths to be simplified
2021-02-16 21:32:28,307 - Identified 8 edge endpoints
2021-02-16 21:32:28,307 - Identified 14 possible paths
2021-02-16 21:32:28,307 - Processing 14 paths
2021-02-16 21:32:28,307 - Found 6 paths to simplify.
2021-02-16 21:32:28,307 - Generated 6 link ids.
2021-02-16 21:32:28,307 - Processing links for all paths to be simplified
2021-02-16 21:32:28,313 - Adding new simplified links
2021-02-16 21:32:28,351 - Generated 0 link ids.
2021-02-16 21:32:28,360 - Added 6 links
2021-02-16 21:32:28,362 - Simplified graph: 14 to 8 nodes, 26 to 14 edges
2021-02-16 21:32:28,362 - Updating the Schedule
2021-02-16 21:32:28,363 - Updated Stop Link Reference Ids
2021-02-16 21:32:28,363 - Updated Network Routes
2021-02-16 21:32:28,363 - Simplification resulted in 18 links being simplified.
2021-02-16 21:32:28,364 - Writing /tmp/genet/network.xml
Traceback (most recent call last):
  File "scripts/simplify_network.py", line 69, in <module>
    n.write_to_matsim(output_dir)
  File "/Users/mickyfitz/workspace/genet/genet/core.py", line 1253, in write_to_matsim
    matsim_xml_writer.write_matsim_network(output_dir, self)
  File "/Users/mickyfitz/workspace/genet/genet/outputs_handler/matsim_xml_writer.py", line 84, in write_matsim_network
    link_attributes = prepare_link_attributes(deepcopy(link_attribs))
  File "/Users/mickyfitz/workspace/genet/genet/outputs_handler/matsim_xml_writer.py", line 58, in prepare_link_attributes
    validate_link_data(link_attributes)
  File "/Users/mickyfitz/workspace/genet/genet/validate/network_validation.py", line 8, in validate_link_data
    raise AttributeError('Attribute: {} missing from link: {}'.format(necessary_link_attrib, link_attributes))
AttributeError: Attribute: oneway missing from link: {'id': '1-3', 'modes': {'cycle', 'car', 'bus'}, 'permlanes': 1.0, 'capacity': 1000.0, 'freespeed': 10.0, 'to': '3', 'from': '1', 'length': 500.0}

After this change

$ python scripts/simplify_network.py -n /Users/mickyfitz/workspace/matesto/data/matsim12-test-town/network.xml -s /Users/mickyfitz/workspace/matesto/data/matsim12-test-town/transitschedule.xml -p epsg:27700 -od /tmp/genet

2021-02-16 21:33:26,026 - Reading in network at /Users/mickyfitz/workspace/matesto/data/matsim12-test-town/network.xml
2021-02-16 21:33:26,029 - Reading in schedule at /Users/mickyfitz/workspace/matesto/data/matsim12-test-town/transitschedule.xml
2021-02-16 21:33:26,443 - Simplifying the Network.
2021-02-16 21:33:26,444 - Begin simplifying the graph
2021-02-16 21:33:26,444 - Generating paths to be simplified
2021-02-16 21:33:26,444 - Identified 8 edge endpoints
2021-02-16 21:33:26,444 - Identified 14 possible paths
2021-02-16 21:33:26,444 - Processing 14 paths
2021-02-16 21:33:26,444 - Found 6 paths to simplify.
2021-02-16 21:33:26,444 - Generated 6 link ids.
2021-02-16 21:33:26,444 - Processing links for all paths to be simplified
2021-02-16 21:33:26,445 - Adding new simplified links
2021-02-16 21:33:26,469 - Generated 0 link ids.
2021-02-16 21:33:26,477 - Added 6 links
2021-02-16 21:33:26,479 - Simplified graph: 14 to 8 nodes, 26 to 14 edges
2021-02-16 21:33:26,479 - Updating the Schedule
2021-02-16 21:33:26,479 - Updated Stop Link Reference Ids
2021-02-16 21:33:26,479 - Updated Network Routes
2021-02-16 21:33:26,479 - Simplification resulted in 18 links being simplified.
2021-02-16 21:33:26,480 - Writing /tmp/genet/network.xml
2021-02-16 21:33:26,710 - Writing /tmp/genet/schedule.xml
2021-02-16 21:33:26,713 - Writing /tmp/genet/vehicles.xml
2021-02-16 21:33:26,717 - Generating validation report
2021-02-16 21:33:26,717 - Checking validity of the Network
2021-02-16 21:33:26,717 - Checking validity of the Network graph
2021-02-16 21:33:26,717 - Checking network connectivity for mode: car
2021-02-16 21:33:26,718 - Checking network connectivity for mode: walk
2021-02-16 21:33:26,718 - Checking network connectivity for mode: bike
2021-02-16 21:33:26,719 - Checking validity of the Schedule
2021-02-16 21:33:26,724 - Graph validation: {'car': {'problem_nodes': {'dead_ends': [], 'unreachable_node': []}, 'number_of_connected_subgraphs': 1}, 'walk': {'problem_nodes': {'dead_ends': [], 'unreachable_node': []}, 'number_of_connected_subgraphs': 0}, 'bike': {'problem_nodes': {'dead_ends': [], 'unreachable_node': []}, 'number_of_connected_subgraphs': 0}}
2021-02-16 21:33:26,724 - Schedule level validation: True
2021-02-16 21:33:26,724 - Routing validation: True
2021-02-16 21:33:26,724 - Generating geojson outputs for the entire network in /tmp/genet/standard_outputs
/Users/mickyfitz/.pyenv/versions/genet-venv-3.7.6/lib/python3.7/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
2021-02-16 21:33:27,231 - Saving network graph nodes and links geojsons to /tmp/genet/standard_outputs
2021-02-16 21:33:27,255 - /tmp/genet/standard_outputs/network_nodes.geojson: No such file or directory
2021-02-16 21:33:27,268 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:27,322 - /tmp/genet/standard_outputs/network_links.geojson: No such file or directory
2021-02-16 21:33:27,322 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:27,353 - /tmp/genet/standard_outputs/network_nodes_geometry_only.geojson: No such file or directory
2021-02-16 21:33:27,354 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:27,374 - /tmp/genet/standard_outputs/network_links_geometry_only.geojson: No such file or directory
2021-02-16 21:33:27,374 - driver GeoJSON does not support creation option ENCODING
/Users/mickyfitz/.pyenv/versions/genet-venv-3.7.6/lib/python3.7/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
2021-02-16 21:33:27,888 - Saving schedule graph nodes and links geojsons to /tmp/genet/standard_outputs
2021-02-16 21:33:27,902 - /tmp/genet/standard_outputs/schedule_nodes.geojson: No such file or directory
2021-02-16 21:33:27,903 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:27,918 - Normalized/laundered field name: 'additional_attributes' to 'additional'
2021-02-16 21:33:27,936 - /tmp/genet/standard_outputs/schedule_links.geojson: No such file or directory
2021-02-16 21:33:27,937 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:27,966 - /tmp/genet/standard_outputs/schedule_nodes_geometry_only.geojson: No such file or directory
2021-02-16 21:33:27,966 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:27,988 - /tmp/genet/standard_outputs/schedule_links_geometry_only.geojson: No such file or directory
2021-02-16 21:33:27,988 - driver GeoJSON does not support creation option ENCODING
/Users/mickyfitz/.pyenv/versions/genet-venv-3.7.6/lib/python3.7/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
2021-02-16 21:33:28,501 - Generating geojson outputs for car/driving modal subgraph
2021-02-16 21:33:28,515 - /tmp/genet/standard_outputs/graph/car_freespeed_subgraph.geojson: No such file or directory
2021-02-16 21:33:28,516 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:28,544 - /tmp/genet/standard_outputs/graph/car_capacity_subgraph.geojson: No such file or directory
2021-02-16 21:33:28,544 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:28,572 - /tmp/genet/standard_outputs/graph/car_permlanes_subgraph.geojson: No such file or directory
2021-02-16 21:33:28,573 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:28,590 - Generating geojson outputs for different highway tags in car modal subgraph
2021-02-16 21:33:28,592 - Generating geometry-only geojson outputs for cycle modal subgraph
2021-02-16 21:33:28,607 - /tmp/genet/standard_outputs/graph/geometry_only_subgraphs/subgraph_geometry_cycle.geojson: No such file or directory
2021-02-16 21:33:28,607 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:28,624 - Generating geometry-only geojson outputs for car modal subgraph
2021-02-16 21:33:28,641 - /tmp/genet/standard_outputs/graph/geometry_only_subgraphs/subgraph_geometry_car.geojson: No such file or directory
2021-02-16 21:33:28,642 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:28,659 - Generating geometry-only geojson outputs for bus modal subgraph
2021-02-16 21:33:28,675 - /tmp/genet/standard_outputs/graph/geometry_only_subgraphs/subgraph_geometry_bus.geojson: No such file or directory
2021-02-16 21:33:28,675 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:28,690 - Generating geojson outputs for schedule
/Users/mickyfitz/.pyenv/versions/genet-venv-3.7.6/lib/python3.7/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
2021-02-16 21:33:29,235 - Generating vehicles per hour for bus
2021-02-16 21:33:29,276 - /tmp/genet/standard_outputs/schedule/vehicles_per_hour/vehicles_per_hour_bus.geojson: No such file or directory
2021-02-16 21:33:29,276 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:29,290 - Normalized/laundered field name: 'from_stop_name' to 'from_stop_'
2021-02-16 21:33:29,290 - Normalized/laundered field name: 'to_stop_name' to 'to_stop_na'
2021-02-16 21:33:29,290 - Normalized/laundered field name: 'service_name' to 'service_na'
2021-02-16 21:33:29,297 - Generating schedule graph for bus
/Users/mickyfitz/.pyenv/versions/genet-venv-3.7.6/lib/python3.7/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
2021-02-16 21:33:29,823 - /tmp/genet/standard_outputs/schedule/subgraphs/schedule_subgraph_links_bus.geojson: No such file or directory
2021-02-16 21:33:29,823 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:29,855 - /tmp/genet/standard_outputs/schedule/subgraphs/schedule_subgraph_nodes_bus.geojson: No such file or directory
2021-02-16 21:33:29,855 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:29,871 - Normalized/laundered field name: 'additional_attributes' to 'additional'
2021-02-16 21:33:29,877 - Saving vehicles per hour for all PT modes
2021-02-16 21:33:29,890 - /tmp/genet/standard_outputs/schedule/vehicles_per_hour/vehicles_per_hour_all_modes.geojson: No such file or directory
2021-02-16 21:33:29,890 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:29,908 - Normalized/laundered field name: 'from_stop_name' to 'from_stop_'
2021-02-16 21:33:29,908 - Normalized/laundered field name: 'to_stop_name' to 'to_stop_na'
2021-02-16 21:33:29,908 - Normalized/laundered field name: 'service_name' to 'service_na'
2021-02-16 21:33:29,917 - Saving vehicles per hour for all PT modes for selected hour slices
2021-02-16 21:33:29,930 - /tmp/genet/standard_outputs/schedule/vehicles_per_hour/vph_all_modes_within_7:30-8:30.geojson: No such file or directory
2021-02-16 21:33:29,931 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:29,943 - Normalized/laundered field name: 'from_stop_name' to 'from_stop_'
2021-02-16 21:33:29,943 - Normalized/laundered field name: 'to_stop_name' to 'to_stop_na'
2021-02-16 21:33:29,944 - Normalized/laundered field name: 'service_name' to 'service_na'
2021-02-16 21:33:29,959 - /tmp/genet/standard_outputs/schedule/vehicles_per_hour/vph_all_modes_within_8:30-9:30.geojson: No such file or directory
2021-02-16 21:33:29,960 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:29,973 - Normalized/laundered field name: 'from_stop_name' to 'from_stop_'
2021-02-16 21:33:29,973 - Normalized/laundered field name: 'to_stop_name' to 'to_stop_na'
2021-02-16 21:33:29,973 - Normalized/laundered field name: 'service_name' to 'service_na'
2021-02-16 21:33:29,991 - /tmp/genet/standard_outputs/schedule/vehicles_per_hour/vph_all_modes_within_17:30-18:30.geojson: No such file or directory
2021-02-16 21:33:29,992 - driver GeoJSON does not support creation option ENCODING
2021-02-16 21:33:30,005 - Normalized/laundered field name: 'from_stop_name' to 'from_stop_'
2021-02-16 21:33:30,005 - Normalized/laundered field name: 'to_stop_name' to 'to_stop_na'
2021-02-16 21:33:30,005 - Normalized/laundered field name: 'service_name' to 'service_na'
2021-02-16 21:33:30,008 - Generating csv for vehicles per hour for each service
2021-02-16 21:33:30,047 - Generating csv for vehicles per hour per stop
2021-02-16 21:33:30,122 - Generating csvs for trips per day
2021-02-16 21:33:30,197 - Finished generating standard outputs. Zipping folder.
2021-02-16 21:33:30,233 - It took 0.001 min to simplify the network.

For discussion

1 - Although this fixes the bug, perhaps it does not go far enough? It feels like maybe we should validate the value of the oneway attribute (it must be '1') when it is present. Where would you do that?

2 - It feels slightly odd that the validation happens at the point we try to write the network, rather than when we read the network in. I guess this is because the typical use case is to read the network, modify it in memory, then write out the new network? And we presumably want users to be able to read in "broken" networks, fix them, and then write them out?

[KasiaKoz]

Awesomesauce

[KasiaKoz]

It feels like maybe we should validate the value of the oneway attribute (it must be '1') when it is present. Where would you do that?

Either at the point of saving to MATSim network as it's a MATSim attribute or in the validation methods which are MATSim specific, though the name of the method is general.. wiki page. It's the only attribute that has a fixed value, so I'd say fine to hardcode it, if there were more I'd say that'd be an additional incentive to using DTDs at the point of save.

we presumably want users to be able to read in "broken" networks, fix them, and then write them out?

yeah, that, and even though genet is very MATSim network inspired/dependent(for now), I don't want it to be. There is no reason for the network to be a valid MATSim network until you try to save it as one.