-
Notifications
You must be signed in to change notification settings - Fork 1
GMNS Network Validator Prompt
General Modeling Network Specification Validation Rules & Auto-Fix Specification
Reference: https://github.com/asu-trans-ai-lab/TAPLite/wiki/Forward-Star-Network-Structure
This document defines the validation rules for GMNS transportation network files. The following principles govern all validation behavior:
- Report all issues. Classify each as Error (must fix), Warning (review), or Info (observation).
- Minimal modification. Only modify files when the fix is unambiguous and specified in the auto-fix rules.
- Never modify coordinates. x_coord and y_coord values are never changed. CRS issues are reported as errors only.
- Generate geometry from node coordinates when geometry column is missing. Use LINESTRING WKT built from the from_node and to_node coordinates. This produces straight-line geometry (acceptable for modeling; curved road shapes require the original extraction tool).
- Preserve original data. Do not drop columns, rename columns, or alter values unless explicitly required by a rule.
- Physical node zone_id must be empty/null. Any physical node with zone_id = 0 must be corrected to null (empty cell in CSV).
- Staged validation pipeline. Stage 1 (Levels 1–5) checks structural correctness, demand consistency, zone connectors, and graph reachability — "can a path exist?" Stage 2 (Level 6) checks path reasonableness from traffic assignment outputs — "is the path plausible?" If Stage 1 fails for an OD pair, Stage 2 is irrelevant for that pair. Do not run ODME (Level 7) or accessibility checks on structurally unreachable ODs.
Every validation run MUST produce the following outputs:
1. validation_report.txt
A plain-text report following the format specified in the Output Format section. This file is always generated, even if no errors are found.
2. Corrected Files (only if modified)
- node_corrected.csv — generated if any node fix was applied (reindexing, zone_id cleanup, etc.)
- link_corrected.csv — generated if any link fix was applied (sorting, geometry generation, VDF fields, etc.)
- demand_corrected.csv — generated only if demand fixes were applied
All corrected files must be provided as downloadable outputs to the user.
3. Zone Connector Report (always generated)
- zone_connector_report.csv — always generated alongside the validation report. See Level 5 for column specification.
4. OD Failure Trace (if unreachable ODs detected)
- od_failure_trace.csv — generated only if unreachable ODs are detected. Shows sample failures with root-cause classification.
The forward star representation requires zone centroids listed first, followed by physical nodes, with strictly increasing IDs throughout.
| Field | Type | Description |
|---|---|---|
| node_id | int | Unique, positive, strictly increasing |
| zone_id | int / NULL | Zone nodes: zone_id = node_id; Physical nodes: must be NULL (empty cell), NOT 0 |
| x_coord | float | Horizontal coordinate (longitude or easting) |
| y_coord | float | Vertical coordinate (latitude or northing) |
- Zone nodes MUST appear first as a contiguous block (zone_id = node_id).
- Physical nodes MUST follow all zone nodes (zone_id = NULL, i.e., empty cell in CSV).
- node_id must be unique, positive, and strictly increasing.
- Physical nodes must NOT have zone_id = 0. If zone_id is 0, it must be corrected to NULL (empty cell).
Triggered if any physical node has zone_id = 0.
For all nodes where zone_id is 0 (and the node is not a zone centroid), set zone_id to NULL (empty cell in CSV output). This is an auto-fix that always applies.
# Python: node.loc[node['zone_id'] == 0, 'zone_id'] = None
# Then write CSV so these appear as empty cells, not "0"REPORT ONLY — never modify coordinates (x_coord, y_coord).
Compare coordinate magnitude ranges between zone nodes and physical nodes:
- If one group has |x_coord| < 360 (degrees) and the other has |x_coord| > 1000 (projected meters/feet), report as ERROR.
- Do NOT assume which CRS is correct. Do NOT convert or modify any coordinates.
Triggered if ANY of the following are true: node_id not unique, node_id not positive, node_id not strictly increasing, or zone block not contiguous.
If triggered:
- Reassign zone nodes → 1..N (preserving original order).
- Reassign physical nodes → N+1..M.
- Update ALL node references in link.csv and demand.csv.
If none triggered → do NOT modify node_id values.
| Field | Type | Description |
|---|---|---|
| link_id | int | Unique (may be reindexed if sorting changes) |
| from_node_id | int | Must exist in node.csv |
| to_node_id | int | Must exist in node.csv |
| length | float | Must be > 0 meters. Round to 2 decimals. |
| free_speed | float/int | Stored in km/h. Required. |
| lanes | int | Number of lanes. Required. |
| capacity | float/int | Link capacity. Required. |
| geometry | string | LINESTRING WKT. Required for spatial representation. |
If geometry column is missing: AUTO-GENERATE from node coordinates.
For each link, construct a straight-line LINESTRING WKT from the from_node coordinates to the to_node coordinates:
geometry = LINESTRING (from_x from_y, to_x to_y)
Important: This produces straight-line geometry between endpoints, which is sufficient for traffic assignment modeling. For accurate curved road shapes, the original network extraction tool (e.g., osm2gmns) must be re-run with geometry output enabled. Log an Info message noting the geometry is straight-line.
Field names are case-sensitive and whitespace-sensitive. Invalid names are treated as missing.
Report as missing if absent (do not generate):
| Column | Description |
|---|---|
| link_type | Link type classification (e.g. motorway, primary, connector) |
| link_type_name | Human-readable link type name |
| dir_flag | Directionality flag (0 = bidirectional, 1 = forward) |
| allowed_uses | Allowed transport modes (e.g. auto, bike, walk) |
Sort by from_node_id ASC, then to_node_id ASC. If already sorted, do NOT rewrite. If unsorted, sort and reassign link_id = 1..L.
A valid connector links: zone node (zone_id ≠ NULL) ↔ physical node (zone_id = NULL). Every zone node must have at least one connector in each direction (inbound and outbound). Report zones without connectors as ERROR with zone_id and missing direction.
Per-class check: If the allowed_uses column exists, connector checks must be performed per allowed_use class (e.g., auto, bike, walk). A zone may have auto connectors but no bike connectors — both must be reported separately.
- Endpoint nodes missing from node.csv → ERROR (do not fix).
- length ≤ 0 → recompute using node coordinates (only if CRS is consistent).
- geometry column missing → auto-generate LINESTRING from node coordinates.
- Link table unsorted → sort and reassign link_id.
NEVER generate link_type — report as error.
Both free_speed and vdf_free_speed_mph must be rounded to the nearest 5:
free_speed = round(free_speed / 5) * 5 # km/h, nearest 5
vdf_free_speed_mph = round((free_speed / 1.60934) / 5) * 5 # mph, nearest 5If the resulting mph is not a multiple of 5, rewrite free_speed so its mph-equivalent rounds correctly. vdf_free_speed_mph must always match the corrected value.
length = round(length, 2) # meters, 2 decimals
vdf_length_mi = round(length / 1609.34, 2) # miles, 2 decimalsUses km/h free_speed:
vdf_fftt = round(((length / 1000) / free_speed) * 60, 2) # minutesIf missing, set defaults:
vdf_alpha = 0.15
vdf_beta = 4.0Range-check only (report as warning, do NOT modify):
- alpha outside [0.1, 2.0]
- beta outside [1.0, 10.0]
| Field | Type | Description |
|---|---|---|
| o_zone_id | int | Must match a zone node_id in node.csv |
| d_zone_id | int | Must match a zone node_id in node.csv |
| volume | float | Must be ≥ 0 |
All referenced zones must exist in node.csv. Every zone in node.csv must have at least one connector link. If valid, no modification.
For each zone that appears in demand.csv, verify it has adequate connectors to carry its demand:
- Production check: If a zone has total outbound demand (sum of volume where o_zone_id = zone) > 0, it MUST have at least one outbound connector (a link FROM this zone centroid TO a physical node). If missing, report as ERROR: "Zone X has production = Y but no outbound connectors."
- Attraction check: If a zone has total inbound demand (sum of volume where d_zone_id = zone) > 0, it MUST have at least one inbound connector (a link FROM a physical node TO this zone centroid). If missing, report as ERROR: "Zone X has attraction = Y but no inbound connectors."
This level diagnoses WHY OD pairs are inaccessible. The goal is to shift debugging from "why does this OD fail?" to "fix these N zones and M ODs resolve."
Generate zone_connector_report.csv with the following columns:
| Column | Description |
|---|---|
| zone_id | Zone identifier |
| centroid_node_id | The node_id of the zone centroid |
| out_conn_count | Number of outbound connectors (links FROM centroid) |
| in_conn_count | Number of inbound connectors (links TO centroid) |
| out_conn_count_auto | Outbound connectors filtered by allowed_uses containing 'auto' (if column exists) |
| in_conn_count_auto | Inbound connectors filtered by allowed_uses containing 'auto' (if column exists) |
| total_production | Sum of demand volume where o_zone_id = zone_id |
| total_attraction | Sum of demand volume where d_zone_id = zone_id |
| attached_physical_nodes | Comma-separated list of physical node IDs connected to this zone |
| flags | Diagnostic flags (see below) |
Flag values for the flags column:
-
NO_OUT_CONN— Zone has zero outbound connectors -
NO_IN_CONN— Zone has zero inbound connectors -
NO_OUT_CONN_AUTO— Zone has zero outbound auto connectors (when allowed_uses exists) -
NO_IN_CONN_AUTO— Zone has zero inbound auto connectors (when allowed_uses exists) -
PROD_NO_OUT— Zone has production > 0 but no outbound connectors -
ATTR_NO_IN— Zone has attraction > 0 but no inbound connectors -
ISOLATED— Zone is not in the largest connected component
Compute connected components (or strongly connected components if the network is directed, i.e., dir_flag is used) on the physical network graph. If allowed_uses column exists, compute components per class (at minimum for the "auto" class).
For each zone centroid, determine which component it belongs to (via its connectors to the physical network).
Summary statistics to report:
- Total number of connected components
- % of zones in the largest component
- List of isolated zones (not in the largest component), sorted by total production + attraction descending
- Top isolated components ranked by total production + attraction
Key insight: If component(origin) != component(destination) for an OD pair, that pair is unreachable by definition. This is often the fastest explanation for large numbers of OD failures.
If any OD pairs in demand.csv are structurally unreachable, generate od_failure_trace.csv with a sample (up to 50 representative failures) showing:
| Column | Description |
|---|---|
| o_zone_id | Origin zone ID |
| d_zone_id | Destination zone ID |
| volume | Demand volume for this OD pair |
| origin_out_connectors | Count of outbound connectors from origin |
| dest_in_connectors | Count of inbound connectors to destination |
| origin_component | Connected component ID of origin |
| dest_component | Connected component ID of destination |
| failure_reason | Root cause classification (see below) |
Failure reason classification (in priority order):
-
NO_ORIGIN_OUT_CONN— Origin zone has no outbound connectors -
NO_DEST_IN_CONN— Destination zone has no inbound connectors -
FILTERED_BY_ALLOWED_USE— Connectors exist but are filtered out by allowed_uses for this demand class -
DIFFERENT_COMPONENTS— Origin and destination are in different connected components -
DIRECTIONALITY— All paths blocked by one-way restrictions (strongly connected component mismatch)
PREREQUISITE: This level requires traffic assignment output files. It is a Stage 2 check — it validates whether routed paths are plausible, not whether paths can exist (that is Stage 1, Levels 1–5).
The user must attach their traffic assignment accessibility outputs. There are four expected files:
| File | Contents | Key Columns |
|---|---|---|
origin_accessibility.csv |
Per-origin summary: how many destinations reachable, avg distance/time | origin_zone_id, total_volume, number_of_destinations, avg_distance_mile, avg_travel_time_min, google_maps_link |
destination_accessibility.csv |
Per-destination summary: how many origins can reach it, avg distance/time | destination_zone_id, total_volume, number_of_origins, avg_distance_mile, avg_travel_time_min, google_maps_link |
inaccessible_od.csv |
List of OD pairs that the assignment engine could not route | o_zone_id, d_zone_id, volume (and any diagnostic columns) |
google_maps_od_distance.csv |
Sampled OD pairs with network path vs. straight-line vs. Google distance | route_key, mode, o_zone_id, d_zone_id, volume, total_distance_mile, straight_line_distance_mile, distance_ratio, total_free_flow_travel_time, google_maps_http_link, WKT_geometry |
If the user mentions accessibility validation, path reasonableness, distance comparison, or asks "why are OD pairs inaccessible" but does NOT attach any of these files, respond with:
"Accessibility validation (Level 6) requires the output files from your traffic assignment run (e.g., DTALite). Please attach the following files to proceed:
1. origin_accessibility.csv 2. destination_accessibility.csv 3. inaccessible_od.csv 4. google_maps_od_distance.csv
These files are typically generated by DTALite or similar traffic assignment engines. You can provide whichever files are available — I will analyze what you provide and note which additional files would help complete the diagnosis."
If the user provides some but not all four files, proceed with analysis on the available files and clearly state which files are missing and what additional diagnostics they would unlock:
- Without origin_accessibility.csv: Cannot identify which origin zones have zero reachable destinations.
- Without destination_accessibility.csv: Cannot identify which destination zones are unreachable from all origins.
- Without inaccessible_od.csv: Cannot enumerate the specific failed OD pairs or their demand volumes. Can still infer from origin/destination files (zones with number_of_destinations = 0 or number_of_origins = 0).
- Without google_maps_od_distance.csv: Cannot perform path reasonableness checks (distance ratio, detour analysis). Stage 2 path validation is skipped.
From origin_accessibility.csv and destination_accessibility.csv, identify:
- Fully inaccessible zones: Zones where number_of_destinations = 0 (as origin) AND number_of_origins = 0 (as destination). These zones are completely disconnected from the network.
- Origin-only inaccessible: Zones where number_of_destinations = 0 but number_of_origins > 0. Suggests missing or misconfigured outbound connectors.
- Destination-only inaccessible: Zones where number_of_origins = 0 but number_of_destinations > 0. Suggests missing or misconfigured inbound connectors.
- Partially connected: Zones where number_of_destinations or number_of_origins is significantly below the network average. Suggests the zone is in a secondary connected component.
Report the total OD impact: how many OD pairs are affected by inaccessible zones (any OD where the origin OR destination is inaccessible).
From google_maps_od_distance.csv, analyze the distance_ratio column (network path distance / straight-line distance):
- Normal range: 1.0 – 2.0. Network paths are typically 20–80% longer than straight-line distance due to road geometry.
- Warning range: 2.0 – 3.0. Possible detour or suboptimal routing. Flag for review.
- Error range: > 3.0. Likely network connectivity issue forcing long detours. May indicate missing links or incorrect one-way coding.
For OD pairs with high distance ratios, provide the Google Maps link for manual verification. The user can click the link to compare the network path with what Google Maps suggests.
If both Stage 1 (network files) and Stage 2 (accessibility outputs) are available, cross-reference findings:
- Zones flagged as inaccessible in Stage 2 SHOULD also appear as problematic in the Stage 1 Zone Connector Report. If a zone is inaccessible in Stage 2 but has connectors in Stage 1, investigate directionality or component isolation.
- Use the google_maps_link column in accessibility files to provide clickable verification links for flagged zones.
If present in link.csv, range-check only (do NOT modify):
- obs_volume ≥ 0
- ref_volume ≥ 0
- coverage ≥ 5%
ODME is controlled by the odme_mode field in settings.csv. By default, odme_mode = 0 (ODME disabled). To run ODME, the user must set odme_mode = 1 in settings.csv.
If the user asks about running ODME or mentions calibrating demand against observed counts, check whether they have a settings.csv file and advise:
"To enable ODME, set
odme_modeto1in your settings.csv file. If you also want VMT (Vehicle Miles Traveled) output, setodme_vmtto1in the same file."
| Field | Default | Set to | Description |
|---|---|---|---|
| odme_mode | 0 | 1 | Enable Origin-Destination Matrix Estimation |
| odme_vmt | 0 | 1 | Enable VMT reporting in ODME output |
Note: ODME cannot fix unreachable ODs; it can only reweight reachable flows to match observed counts. Ensure Stage 1 validation passes before running ODME. If inaccessible zones are detected in Levels 5–6, resolve those first — ODME will silently ignore unreachable OD pairs, producing misleading calibration results.
Modify node.csv ONLY if:
- ID structure invalid (not unique / not positive / not increasing)
- Zone block not contiguous
- Reindexing required
- Physical nodes have zone_id = 0 (must be corrected to NULL/empty)
NEVER modify x_coord or y_coord.
Modify link.csv ONLY if:
- geometry column missing → auto-generate LINESTRING WKT from node coordinates
- Length invalid (≤ 0) and CRS is consistent for recomputation
- Required computable fields missing (vdf_* fields, vdf_alpha, vdf_beta)
- Incorrect speed rounding
- Incorrect attribute formatting
- Unsorted
NEVER generate link_type — report as error.
Modify demand.csv ONLY if:
- Invalid zone IDs
- Negative volumes
- Missing required columns
NEVER modify accessibility output files. These are read-only inputs for diagnostic analysis. Report findings but do not alter origin_accessibility.csv, destination_accessibility.csv, inaccessible_od.csv, or google_maps_od_distance.csv.
The validation_report.txt file must follow this exact structure:
========================================
GMNS NETWORK VALIDATION REPORT
========================================
LEVEL [X] --- [LEVEL NAME]
------------------------------------------
Passed:
- [check description]
Errors:
- [FILE] Row [N]: [FIELD] --- [ISSUE]
Warnings:
- [FILE]: [FIELD] --- [ISSUE]
Stats:
[N nodes (Z zones, P physical), L links, C connectors]
Zone Connectivity Summary:
- Connected components: [N] (largest contains [X]% of zones)
- Zones with no outbound connectors: [list]
- Zones with no inbound connectors: [list]
- Isolated zones (not in largest component): [list]
Accessibility Summary (if Stage 2 files provided):
- Fully inaccessible zones: [N] ([X]%)
- OD pairs affected: [N] ([X]%)
- Missing accessibility files: [list any not provided]
- Distance ratio warnings (>2.0): [N] pairs
- Distance ratio errors (>3.0): [N] pairs
Corrected Files:
- link_corrected.csv
- node_corrected.csv (only if reindexed)
- demand_corrected.csv (only if fixed)
- zone_connector_report.csv
- od_failure_trace.csv (only if failures detected)
Change Summary:
- [description of each change]
Errors Requiring Manual Action:
- [description + remediation guidance]