Merge 81e36f4 into e7cb298

README.md

@@ -39,6 +39,8 @@ There is a general work flow you can follow to get to the point of comparing roa
 
 1. The first step is to collect your data. OS data can be downloaded/purchased from the official [website](https://www.ordnancesurvey.co.uk/business-and-government/products/os-mastermap-highways-network.html) in formats including GML. OSM data can be collected in a number of ways, the method we will focus on is through the website [geofabrik](http://download.geofabrik.de/). From here you can download data for entire countries or cities, in an OSM file format. Once you have downloaded and unzipped your data, you can move onto the next step.
 
+ONLY IN THIS COMMIT 12/11/2018: OSM data first is converted to gpkg format because SQL CASE cannot be implemented when reading pbf format.
+
 2. Once you have your data files, you can place these in the `input` directory. Once there, you can follow the instructions displayed in the `input` directory's `README.md` file in order to convert your files into JSON format and with a correct coordinate projection. Once your files have been successfully converted, you can move onto the next step.
 
 3. Now that you have your data files in a parsable format, you can compare the two data sets to identify common roads between the two. There are two files that make up the OS & OSM Road Comparison Script, `comparator-config.json` and `comparatorY.js`. `comparator-config.json` is where you will edit the I/O settings for the script, telling the script where your input files are, and where you would like it to output results. Once this file has been configured, you can then run the following command in your terminal, while in the root directory.

bin/run.js

@@ -1,8 +1,8 @@
 const {roadFlow}  = require('../src/index.js')
 
 const inputFiles = {
-  "OS": './input/London_OS.gpkg',
-  "OSM": './input/London_OSM.pbf'
+  "OS": './input/OSMM_HIGHWAYS_June18.gpkg',
+  "OSM": './input/UK_OSM.gpkg'
 };
 
 const outputFiles = {

input/README.md

@@ -1,9 +1,10 @@
 # Convert OS Data Steps:
 ### OS Data:
-When converting to JSON should choose the right layer by adding the world `roadlink` for OS data (and `lines` for OSM) to the end of the `org2org` command, the layer *roadlink* will be chosen as JSON file supports only one layer.
+When converting to JSON should choose the right layer by adding the layer name (`roadlink` for OS and `lines` for OSM) to `ogr2org` command, the layer *roadlink* will be chosen as JSON file supports only one layer.
 Download data from company website or server.
 In the terminal run following lines:
-To converts data to JSON format.
+
+To converts data to JSON format use `-f GeoJSON`, `-f ` is used to specify the output format for more choices check official website for `ogr2ogr` command.
 ```
 ogr2ogr -f GeoJSON "./output_file_name.json" "./input_file.ext" roadlink
 ```
@@ -13,7 +14,7 @@ To converts the projection system from UK (EPSG 27700) to International (EPSG 43
 ogr2ogr -f GeoJSON -s_srs "+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.999601 +x_0=400000 +y_0=-100000 +ellps=airy +units=m +no_defs +nadgrids=./OSTN15_NTv2_OSGBtoETRS.gsb" -t_srs EPSG:4326 "./output.json" "./input.json"
 ```
 
-To crop the map to the coordinates required to cover the area of interest.
+To crop the map to the coordinates required to cover the area of interest use `-clipdst Xmin Ymin Xmax Ymax`
 ```
 ogr2ogr —f GeoJSON "./output.json" "./input.json" -clipdst -0.153 51.51 -0.146 51.515
 ```
@@ -24,9 +25,9 @@ ogr2ogr —f GeoJSON "./output.json" "./input.json" -clipdst -0.153 51.51 -0.146
 highwaydedicated, ferry link, ferrynode, hazard, maintenance, reinstatement, restrictionforvehicles, **roadlink**, roadnode, special designation, street, structure, ferryterminal, road, roadjuncion, turnrestriction.
 
 ### OSM Data:
-To convert OS data follow same as with OS Data steps but second step not required because OSM data is in EPSG 4326 projection system, in first step choose `lines` layer and not `roadlink` as OSM does not have the latter layer.
+To convert OS data follow same steps as with OS data but second step not required because OSM data is in EPSG 4326 projection system, in first step choose `lines` layer and not `roadlink` as OSM does not have the latter layer.
 
-### OSM roads:
+### OSM links inside lines layer:
 primary, primary_link, secondary, secondary_link, tertiary, tertiary_link, motorway, motorway_link, trunk, trunk_link, residential, service, living_street, unclassified.
 
 ### Filter OSM links:
@@ -36,16 +37,17 @@ ogr2ogr -f GeoJSON -sql "SELECT * FROM lines WHERE highway in ('motorway', 'trun
 ```
 
 ### For both files (OS and OSM) input files:
-When data converted to JSON, all coordinates should be of arrays of two element arrays (longitude and latitude). However in the conversion process some of the arrays have a sub arrays of coordinates. The user should transform the arrays to arrays of arrays with two elements. This is done by running the script `convert-array.js`. The input and output file are input files to function `process`. This process should be run for both files to avoid the possibility of any incorrectly converted coordinates. This script improved to delete empty entries (coordinates of empty array) in features array because they cause errors when running `comparatorY.js` script.
+When data converted to JSON, all coordinates should be of arrays of two element arrays (longitude and latitude). However in the conversion process some of the arrays have a sub arrays of coordinates, multiLineString. The user should transform the arrays to arrays of arrays with two elements. This is done by running the script `convert-array.js`. The input file name is input paramter of function `process`. This process should be run for both files to avoid the possibility of any incorrectly coordinates. This script improved to delete empty entries (coordinates of empty array) in features array because they cause errors when running `comparator.js` script.
 When running script `convet-array.js` for a file with large size, an error message is displayed:
+
 >FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
-The reason is Node JS default allocated memory is 512mb. The memory can be increased by running **node** command line as follows:
+
+The reason is Node JS default allocated memory size is 512MB. The memory can be increased by running **node** command as follows:
 ```
 node --max-old-space-size=4096 convert-array.js
 ```
 
 The `--max-old-space-size` can have a value of `1024`, `2048`, `3072`, `4096`, `5120`, `6144`, `7168` or `8192`.
-Then the `node comparatorY.js` script can be executed, however when file size is large the command should be `node --max-old-space-size=4096 comparatorY.js`.
 
 # Easier way to get data is to combine the commands as follows:
 ### For OSM run command line:
@@ -61,10 +63,10 @@ ogr2ogr -f GeoJSON -s_srs "+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.999601 +x_0=4000
 ### Reduce Input File
 Run script below to reduce size of input file by choosing only the required information for oneway roads. First script for OS and second for OSM.
 ```
-ogr2ogr -f GeoJSON -sql "SELECT localid,roadname,directionality,formofway FROM roadlink" "./lonOS.json" "./londonOS.json"
+ogr2ogr -f GeoJSON -sql "SELECT localid, roadname, directionality, formofway FROM roadlink" "./lonOS.json" "./londonOS.json"
 ```
 ```
-ogr2ogr -f GeoJSON -sql "SELECT osm_id,name,other_tags FROM lines" "./lonOSM.json" "./londonOSM.json"
+ogr2ogr -f GeoJSON -sql "SELECT osm_id, name, other_tags FROM lines" "./lonOSM.json" "./londonOSM.json"
 ```
 
 # Script (map-splitter.js): Run ogr2ogr Automatically

src/comparator/comparator.js

@@ -2,7 +2,8 @@
 
 const io = require('./io.js');
 const {calculateProgress} = require('./progress.js');
-const {compareOSroadWithOSM}= require('./compareOSroadWithOSM.js');
+const {compareOSroadWithOSM} = require('./compareOSroadWithOSM.js');
+const {filterDuplication} = require('./filter-output.js');
 const print = require('./print.js');
 const startTime = new Date();
 
@@ -17,15 +18,15 @@ const outputResults = (outputFiles, outputData, roadCounters) => { //function to
 
 //compare names of the roads for match betwwen OS and OSM
 exports.compareData = (input, outputFiles) => {
-  let roadCounters = {  //object holds counter of road links
+  const roadCounters = {  //object holds counter of road links
     noMatch: 0,  //counter of zero match
     oneMatch: 0,  //counter of one match
     multiMatch: 0,  //counter of multi match
     noName: 0,  //counter of no names links
     processedOS: 0,  //counter for processed OS links
     totalRoadsOS: 0  //counter for total links in OS
   };
-  let outputData= { OS: [], OSM: [], info: []}; //object of 3 output data arrays
+  const outputData= { OS: [], OSM: [], info: []}; //object of 3 output data arrays
   [dataOS, dataOSM] = io.read(input); //read input files
   print.header(dataOS.features.length, dataOSM.features.length); //print total links in OS and OSM
   roadCounters.totalRoadsOS = dataOS.features.length; //save total number of OS links to counter
@@ -36,9 +37,10 @@ exports.compareData = (input, outputFiles) => {
       roadCounters.noName ++;
       continue; //no comparision, continue loop
     }
-    let key = compareOSroadWithOSM(roadOS, dataOSM, outputData); //compare OS link against OSM links
+    const key = compareOSroadWithOSM(roadOS, dataOSM, outputData); //compare OS link against OSM links
     roadCounters[key] ++; // increament related counter according to key value
     print.progress(calculateProgress(roadCounters)); //print the progess of the calculation on the run
   }
+  outputData.OS = filterDuplication(outputData.OS);
   outputResults(outputFiles, outputData, roadCounters); //call inner function
 }