Merge pull request #262 from Open-Telecoms-Data/wkt

Use WKT for node locations and span routes

docs/history/changelog.md

@@ -32,6 +32,10 @@ Iterative improvements are made outside of the release cycle. They do not involv
 
 - [#265](https://github.com/Open-Telecoms-Data/open-fibre-data-standard/pull/265) - Update external codelists.
 
+### Normative documentation
+
+- [#262](https://github.com/Open-Telecoms-Data/open-fibre-data-standard/pull/262) - Use well-known text to represent geometries in the CSV publication format.
+
 ### Non-normative documentation
 
 - [#259](https://github.com/Open-Telecoms-Data/open-fibre-data-standard/pull/259) - Fix link to Lib CoVE OFDS

docs/reference/publication_formats/csv.md

@@ -8,9 +8,9 @@ We want to hear your feedback on the standard and its documentation. For general
 To comment on or create discussions and issues, you need to [sign up for a free GitHub account](https://github.com/signup). If you prefer to provide feedback privately, you can email [info@opentelecomdata.net](mailto:info@opentelecomdata.net).
 ```
 
-This section describes the structure of the tables in the CSV format and the relationship between the tables. Example CSV files and blank templates are provided for each table.
+The CSV format consists of several tables, reflecting the structure of the [schema](../schema.md). This pages describes the CSV publication format, including the structure of the tables, the relationship between the tables and the representation of location data. Example CSV files and blank templates are provided for each table.
 
-The CSV format consists of several tables, reflecting the structure of the [schema](../schema.md). The networks table is the main table. Arrays of objects in the schema are represented as separate tables:
+The networks table is the main table. Arrays of objects in the schema are represented as separate tables:
 
 ```{contents}
 ---
@@ -54,6 +54,29 @@ The following example shows a network with two nodes represented in JSON format
 
 ::::
 
+Node locations and span routes are represented using [well-known text (WKT)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) `POINT` and `LINESTRING` geometries, respectively. For example:
+
+::::{tab-set}
+
+:::{tab-item} Node locations
+
+| nodes/0/location |
+| --- |
+| POINT (-0.174 5.625) |
+| POINT (-1.628 6.711) |
+
+:::
+
+:::{tab-item} Span routes
+
+| spans/0/route |
+| --- |
+| LINESTRING (-0.173 5.626,-0.178 5.807,-0.112 5.971,-0.211 5.963,-0.321 6.17,-0.488 6.29,-0.56 6.421,-0.752 6.533,-0.867 6.607,-1.101 6.585,-1.304 6.623,-1.461 6.727,-1.628 6.713) |
+
+:::
+
+::::
+
 ## networks
 
 This table is related to the following tables:
@@ -69,6 +92,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,name,website,publisher,publicationDate,collectionDate,crs,accuracy,accuracyDetails,language
+:nocrossref:
 ```
 
 ### nodes
@@ -83,6 +107,8 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,nodes/0/id,nodes/0/name,nodes/0/phase,nodes/0/status,nodes/0/location,nodes/0/address,nodes/0/type,nodes/0/accessPoint,nodes/0/power,nodes/0/technologies,nodes/0/physicalInfrastructureProvider
+:collapse: nodes/0/location
+:nocrossref:
 ```
 
 #### nodes_internationalConnections
@@ -95,6 +121,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,nodes/0/id,nodes/0/internationalConnections/0/streetAddress,nodes/0/internationalConnections/0/locality,nodes/0/internationalConnections/0/region,nodes/0/internationalConnections/0/postalCode,nodes/0/internationalConnections/0/country
+:nocrossref:
 ```
 
 #### nodes_networkProviders
@@ -107,6 +134,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,nodes/0/id,nodes/0/networkProviders/0/id,nodes/0/networkProviders/0/name
+:nocrossref:
 ```
 
 ### spans
@@ -120,6 +148,8 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,spans/0/id,spans/0/name,spans/0/phase,spans/0/status,spans/0/readyForServiceDate,spans/0/start,spans/0/end,spans/0/directed,spans/0/route,spans/0/physicalInfrastructureProvider,spans/0/supplier,spans/0/transmissionMedium,spans/0/deployment,spans/0/deploymentDetails,spans/0/darkFibre,spans/0/fibreType,spans/0/fibreTypeDetails,spans/0/fibreCount,spans/0/fibreLength,spans/0/technologies,spans/0/capacity,spans/0/capacityDetails,spans/0/countries
+:collapse: spans/0/route
+:nocrossref:
 ```
 
 #### spans_networkProviders
@@ -132,6 +162,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,spans/0/id,spans/0/networkProviders/0/id,spans/0/networkProviders/0/name
+:nocrossref:
 ```
 
 ### phases
@@ -145,6 +176,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,phases/0/id,phases/0/name,phases/0/description
+:nocrossref:
 ```
 
 #### phases_funders
@@ -157,6 +189,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,phases/0/id,phases/0/funders/0/id,phases/0/funders/0/name
+:nocrossref:
 ```
 
 ### organisations
@@ -169,6 +202,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,organisations/0/id,organisations/0/name,organisations/0/identifier,organisations/0/country,organisations/0/roles,organisations/0/roleDetails,organisations/0/website,organisations/0/logo
+:nocrossref:
 ```
 
 ### contracts
@@ -183,6 +217,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,contracts/0/id,contracts/0/title,contracts/0/description,contracts/0/type,contracts/0/value,contracts/0/dateSigned
+:nocrossref:
 ```
 
 #### contracts_documents
@@ -195,6 +230,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,contracts/0/id,contracts/0/documents/0/title,contracts/0/documents/0/description,contracts/0/documents/0/url,contracts/0/documents/0/format
+:nocrossref:
 ```
 
 #### contracts_relatedPhases
@@ -207,6 +243,7 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,contracts/0/id,contracts/0/relatedPhases/0/id,contracts/0/relatedPhases/0/name
+:nocrossref:
 ```
 
 ### links
@@ -219,4 +256,5 @@ The fields in this table are listed below. You can also download an [example CSV
 
 ```{jsonschema} ../../../schema/network-schema.json
 :include: id,links/0/href,links/0/rel
+:nocrossref:
 ```

examples/csv/nodes.csv

@@ -1,3 +1,3 @@
-id,nodes/0/id,nodes/0/name,nodes/0/phase/id,nodes/0/phase/name,nodes/0/status,nodes/0/location/type,nodes/0/location/coordinates,nodes/0/address/streetAddress,nodes/0/address/locality,nodes/0/address/region,nodes/0/address/postalCode,nodes/0/address/country,nodes/0/type,nodes/0/accessPoint,nodes/0/power,nodes/0/technologies,nodes/0/physicalInfrastructureProvider/id,nodes/0/physicalInfrastructureProvider/name
-a096d627-72e1-4f9b-b129-951b1737bff4,1,Accra,1,NextGen Phase 1,operational,Point,-0.174;5.625,Boundary Rd,Shiashie,Accra,12345,GH,pop,True,True,ip,GH-RGD-CS111111111,FibreCo
-a096d627-72e1-4f9b-b129-951b1737bff4,2,Kumasi,1,NextGen Phase 1,operational,Point,-1.628;6.711,Kejetia Road,Bantama,Kumasi,00000,GH,pop,False,True,ip,GH-RGD-CS111111111,FibreCo
+id,nodes/0/id,nodes/0/name,nodes/0/phase/id,nodes/0/phase/name,nodes/0/status,nodes/0/location,nodes/0/address/streetAddress,nodes/0/address/locality,nodes/0/address/region,nodes/0/address/postalCode,nodes/0/address/country,nodes/0/type,nodes/0/accessPoint,nodes/0/power,nodes/0/technologies,nodes/0/physicalInfrastructureProvider/id,nodes/0/physicalInfrastructureProvider/name
+a096d627-72e1-4f9b-b129-951b1737bff4,1,Accra,1,NextGen Phase 1,operational,POINT (-0.174 5.625),Boundary Rd,Shiashie,Accra,12345,GH,pop,True,True,ip,GH-RGD-CS111111111,FibreCo
+a096d627-72e1-4f9b-b129-951b1737bff4,2,Kumasi,1,NextGen Phase 1,operational,POINT (-1.628 6.711),Kejetia Road,Bantama,Kumasi,00000,GH,pop,False,True,ip,GH-RGD-CS111111111,FibreCo

examples/csv/spans.csv

@@ -1,2 +1,2 @@
-id,spans/0/id,spans/0/name,spans/0/phase/id,spans/0/phase/name,spans/0/status,spans/0/readyForServiceDate,spans/0/start,spans/0/end,spans/0/directed,spans/0/route/type,spans/0/route/coordinates,spans/0/physicalInfrastructureProvider/id,spans/0/physicalInfrastructureProvider/name,spans/0/supplier/id,spans/0/supplier/name,spans/0/transmissionMedium,spans/0/deployment,spans/0/deploymentDetails/description,spans/0/darkFibre,spans/0/fibreType,spans/0/fibreTypeDetails/fibreSubtype,spans/0/fibreTypeDetails/description,spans/0/fibreCount,spans/0/fibreLength,spans/0/technologies,spans/0/capacity,spans/0/capacityDetails/description,spans/0/countries
-a096d627-72e1-4f9b-b129-951b1737bff4,1,Accra to Kumasi,1,NextGen Phase 1,operational,2018-01-01,1,2,False,LineString,"-0.173,5.626;-0.178,5.807;-0.112,5.971;-0.211,5.963;-0.321,6.17;-0.488,6.29;-0.56,6.421;-0.752,6.533;-0.867,6.607;-1.101,6.585;-1.304,6.623;-1.461,6.727;-1.628,6.713",GH-RGD-CS111111111,FibreCo,GH-RGD-CS333333333,ProjectCo,fibre,belowGround,Buried in ducts,True,G.652,G.652.D,"CABLED STANDARD ENHANCED SINGLEMODE FIBRE ESMF, LOW WATER PEAK SINGLE MODE FIBRE G.652.D, OS2",24,276000,sdh,4.976,2xSTM-16,GH
+id,spans/0/id,spans/0/name,spans/0/phase/id,spans/0/phase/name,spans/0/status,spans/0/readyForServiceDate,spans/0/start,spans/0/end,spans/0/directed,spans/0/route,spans/0/physicalInfrastructureProvider/id,spans/0/physicalInfrastructureProvider/name,spans/0/supplier/id,spans/0/supplier/name,spans/0/transmissionMedium,spans/0/deployment,spans/0/deploymentDetails/description,spans/0/darkFibre,spans/0/fibreType,spans/0/fibreTypeDetails/fibreSubtype,spans/0/fibreTypeDetails/description,spans/0/fibreCount,spans/0/fibreLength,spans/0/technologies,spans/0/capacity,spans/0/capacityDetails/description,spans/0/countries
+a096d627-72e1-4f9b-b129-951b1737bff4,1,Accra to Kumasi,1,NextGen Phase 1,operational,2018-01-01,1,2,False,"LINESTRING (-0.173 5.626,-0.178 5.807,-0.112 5.971,-0.211 5.963,-0.321 6.17,-0.488 6.29,-0.56 6.421,-0.752 6.533,-0.867 6.607,-1.101 6.585,-1.304 6.623,-1.461 6.727,-1.628 6.713)",GH-RGD-CS111111111,FibreCo,GH-RGD-CS333333333,ProjectCo,fibre,belowGround,Buried in ducts,True,G.652,G.652.D,"CABLED STANDARD ENHANCED SINGLEMODE FIBRE ESMF, LOW WATER PEAK SINGLE MODE FIBRE G.652.D, OS2",24,276000,sdh,4.976,2xSTM-16,GH

examples/csv/template/nodes.csv

@@ -1 +1 @@
-id,nodes/0/id,nodes/0/name,nodes/0/phase/id,nodes/0/phase/name,nodes/0/status,nodes/0/location/type,nodes/0/location/coordinates,nodes/0/address/streetAddress,nodes/0/address/locality,nodes/0/address/region,nodes/0/address/postalCode,nodes/0/address/country,nodes/0/type,nodes/0/accessPoint,nodes/0/power,nodes/0/technologies,nodes/0/physicalInfrastructureProvider/id,nodes/0/physicalInfrastructureProvider/name
+id,nodes/0/id,nodes/0/name,nodes/0/phase/id,nodes/0/phase/name,nodes/0/status,nodes/0/location,nodes/0/address/streetAddress,nodes/0/address/locality,nodes/0/address/region,nodes/0/address/postalCode,nodes/0/address/country,nodes/0/type,nodes/0/accessPoint,nodes/0/power,nodes/0/technologies,nodes/0/physicalInfrastructureProvider/id,nodes/0/physicalInfrastructureProvider/name

examples/csv/template/spans.csv

@@ -1 +1 @@
-id,spans/0/id,spans/0/name,spans/0/phase/id,spans/0/phase/name,spans/0/status,spans/0/readyForServiceDate,spans/0/start,spans/0/end,spans/0/directed,spans/0/route/type,spans/0/route/coordinates,spans/0/physicalInfrastructureProvider/id,spans/0/physicalInfrastructureProvider/name,spans/0/supplier/id,spans/0/supplier/name,spans/0/transmissionMedium,spans/0/deployment,spans/0/deploymentDetails/description,spans/0/darkFibre,spans/0/fibreType,spans/0/fibreTypeDetails/fibreSubtype,spans/0/fibreTypeDetails/description,spans/0/fibreCount,spans/0/fibreLength,spans/0/technologies,spans/0/capacity,spans/0/capacityDetails/description,spans/0/countries
+id,spans/0/id,spans/0/name,spans/0/phase/id,spans/0/phase/name,spans/0/status,spans/0/readyForServiceDate,spans/0/start,spans/0/end,spans/0/directed,spans/0/route,spans/0/physicalInfrastructureProvider/id,spans/0/physicalInfrastructureProvider/name,spans/0/supplier/id,spans/0/supplier/name,spans/0/transmissionMedium,spans/0/deployment,spans/0/deploymentDetails/description,spans/0/darkFibre,spans/0/fibreType,spans/0/fibreTypeDetails/fibreSubtype,spans/0/fibreTypeDetails/description,spans/0/fibreCount,spans/0/fibreLength,spans/0/technologies,spans/0/capacity,spans/0/capacityDetails/description,spans/0/countries

manage.py

@@ -207,6 +207,20 @@ def generate_csv_reference_markdown(table, schema, parents=None, depth=2):
   markdown[table]['content'].extend([
     "```{jsonschema} ../../../schema/network-schema.json\n"
     f":include: {','.join(include_pointers)}\n"
+  ])
+
+  # Collapse node locations and span routes, which are represented using well-known text in the CSV format
+  if table == 'nodes':
+    markdown[table]['content'].extend([
+      ":collapse: nodes/0/location\n"
+    ])
+  elif table == 'spans':
+    markdown[table]['content'].extend([
+      ":collapse: spans/0/route\n"
+    ])
+
+  markdown[table]['content'].extend([
+    ":nocrossref:\n"
     "```\n"
   ])
 
@@ -535,6 +549,35 @@ def pre_commit():
       line_terminator='LF',
     )
 
+    # Use WKT geometry in CSV examples and templates. This code should be removed once Flatten Tool supports WKT
+    replacements = {
+       'examples/csv/nodes.csv': [
+          ('nodes/0/location/type,nodes/0/location/coordinates','nodes/0/location'),
+          ('Point,-0.174;5.625', 'POINT (-0.174 5.625)'),
+          ('Point,-1.628;6.711', 'POINT (-1.628 6.711)')
+       ],
+       'examples/csv/template/nodes.csv': [
+          ('nodes/0/location/type,nodes/0/location/coordinates','nodes/0/location')
+       ],
+       'examples/csv/spans.csv': [
+          ('spans/0/route/type,spans/0/route/coordinates','spans/0/route'),
+          ('LineString,"-0.173,5.626;-0.178,5.807;-0.112,5.971;-0.211,5.963;-0.321,6.17;-0.488,6.29;-0.56,6.421;-0.752,6.533;-0.867,6.607;-1.101,6.585;-1.304,6.623;-1.461,6.727;-1.628,6.713"', '"LINESTRING (-0.173 5.626,-0.178 5.807,-0.112 5.971,-0.211 5.963,-0.321 6.17,-0.488 6.29,-0.56 6.421,-0.752 6.533,-0.867 6.607,-1.101 6.585,-1.304 6.623,-1.461 6.727,-1.628 6.713)"')
+       ],
+       'examples/csv/template/spans.csv': [
+          ('spans/0/route/type,spans/0/route/coordinates','spans/0/route')
+       ]
+    }
+
+    for key, value in replacements.items():
+      with open(key, 'r') as f:
+         content = f.read()
+
+      for replacement in value:
+         content = content.replace(replacement[0], replacement[1])
+
+      with open(key, 'w') as f:
+         f.write(content)
+
     # Update docs/reference/publication_formats/csv.md
     update_csv_docs(jsonref_schema)
 

schema/network-schema.csv

@@ -15,7 +15,7 @@ nodes,nodes/phase,Phase reference,The identifier and name of the phase being ref
 nodes,nodes/phase/id,Identifier,The identifier of the phase being referenced. This must match the `.id` of a phase in the `phases` array.,string,1..1,,,,,
 nodes,nodes/phase/name,Name,The name of the phase being referenced. This must match the `.name` of the phase in the `phases` array whose `.id` matches the `.id` of this phase reference.,string,0..1,,,,,
 nodes,nodes/status,Status,"The status of this node, from the closed nodeStatus codelist.",string,0..1,"Enum: proposed, planned, underConstruction, operational, decommissioned, inactive",https://open-fibre-data-standard.readthedocs.io/en/latest/reference/codelists.html#nodestatus,,,nodeStatus.csv
-nodes,nodes/location,Node location,A GeoJSON Point geometry describing the physical location of this node.,object,0..1,,https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.4,,,
+nodes,nodes/location,Node location,"In the JSON publication format, a GeoJSON Point geometry describing the physical location of this node. In the CSV publication format, a well-known text POINT geometry.",object,0..1,,"https://open-fibre-data-standard.readthedocs.io/en/latest/reference/publication_formats/json.html, https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.4, https://open-fibre-data-standard.readthedocs.io/en/latest/reference/publication_formats/csv.html, https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry",,,
 nodes,nodes/location,Geometry,A GeoJSON Geometry object.,object,0..1,,https://datatracker.ietf.org/doc/html/rfc7946#section-3.1,,,
 nodes,nodes/location/type,Type,"The GeoJSON geometry type that is described by `.coordinates`, from the closed geometryType codelist. This must be 'Point' when referenced by `Node.location`, and 'LineString' when referenced by `Span.route`.",string,1..1,"Enum: LineString, Point","https://datatracker.ietf.org/doc/html/rfc7946#section-1.4, https://open-fibre-data-standard.readthedocs.io/en/latest/reference/codelists.html#geometrytype",,,geometryType.csv
 nodes,nodes/location/coordinates,Coordinates,"One or more GeoJSON positions. For geometries of type 'Point', the coordinates member must be a single position. For geometries of type 'LineString' the coordinates member must be an array of positions.
@@ -67,7 +67,7 @@ spans,spans/readyForServiceDate,Ready for service date,"The date this span was r
 spans,spans/start,Start,"The identifier of one of two nodes that this span connects. It must match the `.id` of a node in the `nodes` array. If `directed == true`, it represents the starting point of the span. Otherwise, the distinction between `start` and `end` is arbitrary.",string,0..1,,,,,
 spans,spans/end,End,"The identifier of one of two nodes that this span connects. It must match the `.id` of a node in the `nodes` array. If `directed == true`, it represents the end point of the span. Otherwise, the distinction between `start` and `end` is arbitrary.",string,0..1,,,,,
 spans,spans/directed,Directed,"Whether this span is directed. If `True`, then `.start` represents the start of the span and `.end` represents the end. Otherwise, `.start` and `.end` reference this span's endpoints with no direction implied.",boolean,0..1,,,,,
-spans,spans/route,Span route,A GeoJSON LineString geometry describing the route of this span.,object,0..1,,https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.4,,,
+spans,spans/route,Span route,"In the JSON publication format, a GeoJSON LineString geometry describing the route of this span. In the CSV publication format, a well-known text LINESTRING geometry.",object,0..1,,"https://open-fibre-data-standard.readthedocs.io/en/latest/reference/publication_formats/json.html, https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.4, https://open-fibre-data-standard.readthedocs.io/en/latest/reference/publication_formats/csv.html, https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry",,,
 spans,spans/route,Geometry,A GeoJSON Geometry object.,object,0..1,,https://datatracker.ietf.org/doc/html/rfc7946#section-3.1,,,
 spans,spans/route/type,Type,"The GeoJSON geometry type that is described by `.coordinates`, from the closed geometryType codelist. This must be 'Point' when referenced by `Node.location`, and 'LineString' when referenced by `Span.route`.",string,1..1,"Enum: LineString, Point","https://datatracker.ietf.org/doc/html/rfc7946#section-1.4, https://open-fibre-data-standard.readthedocs.io/en/latest/reference/codelists.html#geometrytype",,,geometryType.csv
 spans,spans/route/coordinates,Coordinates,"One or more GeoJSON positions. For geometries of type 'Point', the coordinates member must be a single position. For geometries of type 'LineString' the coordinates member must be an array of positions.