Command refactoring and more

Makefile.am

@@ -70,8 +70,8 @@ help:
 	@echo "services-docs           - If configure '--with-services', then builds just the web services documentation."
 	@echo "services-test           - If configure '--with-services', then runs the web services slow tests.  Should take five to ten minutes."
 	@echo "services-test-all       - If configure '--with-services', then runs the web services glacial tests.  Should take five to ten minutes."
-	@echo "test                    - Run slow tests.  Should take five to ten minutes."
-	@echo "test-all                - Run all tests.  Could take hours."
+	@echo "test                    - Run slow tests.  Should take no more than five to ten minutes."
+	@echo "test-all                - Run all tests.  Could take more than ten minutes."
 	@echo "test-quick              - Run quick tests.  Should take less than five minutes."
 	@echo "ui-build                - Build the web UI js/css dist.  Must enable --with-services configuration options."
 	@echo "ui-test                 - Run user interface tests.  May take several minutes.  Must enable --with-uitests and --with-services configuration options."

README.md

@@ -63,6 +63,7 @@ typing 'hoot' at the command line in the Hootenanny environment.  More detail is
 
 # Supported Data Formats
 **Hootenanny can import from:**
+* OGR supported formats
 * ESRI File Geodatabase (.gdb)
 * GeoJSON (.geojson) (M)
 * geonames.org (.geonames)
@@ -73,9 +74,9 @@ typing 'hoot' at the command line in the Hootenanny environment.  More detail is
 * OpenStreetMap API Database (osmapidb://)
 * Shapefile (.shp)
 * Zip files containing shapefiles and/or ESRI File Geodatabase files (.zip)
-* OGR supported formats
 
 **Hootenanny can export to:** 
+* OGR supported formats
 * ESRI File Geodatabase (.gdb)
 * GeoJSON (.geojson) (M)
 * Hootenanny API Database (hootapidb://)
@@ -84,7 +85,6 @@ typing 'hoot' at the command line in the Hootenanny environment.  More detail is
 * OpenStreetMap Protocol Buffers file (.osm.pbf)
 * OpenStreetMap API Database (osmapidb://)
 * Shapefile (.shp) (M)
-* OGR supported formats
 
 **Hootenanny can export changesets to:** 
 * OpenStreetMap XML changeset file (.osc) (M)

conf/core/ConfigOptions.asciidoc

@@ -12,7 +12,7 @@
 // * What does this configuration change?
 // * If relevant, what are the units of the config option (e.g. meters)?
 // * Where can the user go to get more information? Please reference the
-//   appropriate document. E.g. "See `convert-ogr2osm` in the _Command Line
+//   appropriate document. E.g. "See `convert` in the _Command Line
 //   Reference_ for more information."
 // * What other options may be of interest? E.g. "See also
 //   `big.perty.op.sigma`"
@@ -484,8 +484,8 @@ the convert.bounding.box setting for OSM API database data sources only.
 * Default Value:
 ** ``
 
-Specifies one or more semi-colon delimited operations to perform before writing data. This is only
-applicable to the convert command.
+Specifies one or more semi-colon delimited operations to perform before writing converted data.
+This is only applicable to the convert command.
 
 === cookie.cutter.alpha
 
@@ -830,7 +830,7 @@ of assigning IDs within OSM.
 Example Usage:
 
 ----
-hoot convert-ogr2osm -D id.generator=hoot::PositiveIdGenerator -D id.generator.node.start=100 -D id.generator.relation.start=200 -D id.generator.way.start=300 $HOOT_HOME/translations/Identity.js myoutput.osm myinput.osm
+hoot convert -D id.generator=hoot::PositiveIdGenerator -D id.generator.node.start=100 -D id.generator.relation.start=200 -D id.generator.way.start=300 myinput.osm myoutput.osm --trans $HOOT_HOME/translations/Identity.js
 ----
 
 === id.generator.node.start
@@ -1255,7 +1255,7 @@ RAM usage of applications in a shared server environment.
 * Data Type: double
 * Default Value: `1.0`
 
-When merging nodes during convert-ogr2osm, determines what tolerance should be used for deciding two nodes
+When merging nodes during convert, determines what tolerance should be used for deciding two nodes
 are identical. Units in meters and defaults to 1.0m.
 
 [[merger.creators]]
@@ -1528,7 +1528,7 @@ intelligently use indexes, others will simply filter features after reading from
 Example Usage:
 
 ----
-hoot convert-ogr2osm -D ogr.reader.bounding.box=106.851,-6.160,107.052,-5.913 translations/Identity.js output.osm test-files/jakarta_raya_coastline.shp
+hoot convert -D ogr.reader.bounding.box=106.851,-6.160,107.052,-5.913 test-files/jakarta_raya_coastline.shp output.osm --trans translations/Identity.js
 ----
 
 === ogr.reader.bounding.box.latlng
@@ -1645,14 +1645,20 @@ hoot convert \
   PG:"dbname='osm_gis2' host='localhost' port='5432' user='hoot' password='hoottest'"
 ----
 
-=== ogr2osm.ops
+=== ogr2osm.merge.nearby.nodes
 
-* Data Type: list
-* Default Value:
-** `hoot::MergeNearbyNodes` - Merges nodes within the distance defined by merge.nearby.nodes.distance.
-** `hoot::BuildingPartMergeOp` - Implicitly merges certain individual building parts into a single part.
+* Data Type: bool
+* Default Value: `true`
+
+Merges nearby nodes together when converting from an OGR format to OSM.
+
+=== ogr2osm.simplify.complex.buildings
+
+* Data Type: bool
+* Default Value: `true`
 
-Operations that should be applied to ingested OGR data before saving the data.
+Implicitly merges certain individual building parts into a single part when converting from an OGR
+format to OSM.
 
 === osmapidb.bulk.inserter.disable.database.constraints.during.write
 
@@ -1748,14 +1754,6 @@ written if the value is not empty.
 If true, the OSM map writer will NOT write a file if the map is empty.
 The default is to write a file even if the map is empty.
 
-=== osm2ogr.ops
-
-* Data Type: list
-* Default Value:
-** ``
-
-User specified operations for converting OSM files to OGR compatible file types.
-
 === perty.apply.rubber.sheet
 
 * Data Type: bool
@@ -2917,7 +2915,7 @@ NOTE: This assumes that you know exactly what tags you want to modify/delete
 * Data Type: string
 * Default Value: ``
 
-The script to use for translation.
+The script used by the TranslationVisitor for translation.
 
 === unify.enable.optimal.constrained.matches
 
@@ -2958,7 +2956,6 @@ before the operations in `conflate.post.ops`.
 * Default Value:
 ** ``
 
-
 List of operations to apply immediately before conflating when using Unifying Conflation only.  Runs
 after the operations in `conflate.pre.ops`.
 

conf/core/RemoveReview2Pre.conf

@@ -1,11 +1,8 @@
 {
-  "#" : "Options to remove all to-be-reviewed elements before conflation, convert or osm2ogr.",
-  "osm2ogr.ops": "hoot::RemoveElementsVisitor",
-  "unify.pre.ops": "$(unify.pre.ops);hoot::RemoveElementsVisitor",
+  "#" : "Options to remove all to-be-reviewed elements before conflation",
   "convert.ops": "hoot::RemoveElementsVisitor",
+  "unify.pre.ops": "$(unify.pre.ops);hoot::RemoveElementsVisitor",
   "remove.elements.visitor.filter": "hoot::TagCriterion",
   "tag.criterion.kvp": "hoot:review:source,2",
-
-
   "#" : "end"
 }

conf/core/TdsExport.json

@@ -3,10 +3,9 @@
   "unify.post.ops": "hoot::RemoveElementsVisitor",
   "remove.elements.visitor.filter": "hoot::TagFilter",
   "tag.filter.kvp": "hoot:review:source,2",
-  "osm2ogr.ops": "$(osm2ogr.ops);hoot::SmallWayMerger",
+  "convert.ops": "$(convert.ops);hoot::SmallWayMerger",
   "small.way.merger.diff": "hoot::TranslatedTagDifferencer",
   "translated.tag.differencer.script": "$(HOOT_HOME)/translations/TDSv40.js",
   "translated.tag.differencer.ignore.list": "UFI",
-
   "#" : "end"
 }

conf/services/conflateAdvOps.json

@@ -349,7 +349,7 @@
             	"id":"merge_nearby_nodes_distance",
                 "minvalue":"0.01",
                 "elem_type":"double",
-                "description":"When merging nodes during ogr2osm, determines what tolerance should be used for deciding two nodes are identical. Units in meters and defaults to 1.0m.",
+                "description":"When merging nodes during convert, determines what tolerance should be used for deciding two nodes are identical. Units in meters and defaults to 1.0m.",
                 "maxvalue":"",
                 "name":"Merge Nearby Nodes Distance",
                 "defaultvalue":"1.0",

docs/HootenannyConfigOptions.asciidoc

@@ -77,7 +77,7 @@ Configuration parameters:
 [[OGR-Writer]]
 ==== OGR Writer
 
-The OGR Writer supports advanced translation of OSM formatted files into well defined tabular formats such as Shapefile and FileGDB. See the +--convert-osm2ogr+ documentation for specific examples of how to use the OGR Writer.
+The OGR Writer supports advanced translation of OSM formatted files into well defined tabular formats such as Shapefile and FileGDB. See the +--convert+ documentation for specific examples of how to use the OGR Writer.
 
 Configuration Parameters:
 
@@ -102,11 +102,10 @@ hoot convert \
   PG:"dbname='osm_gis2' host='localhost' port='5432' user='hoot' password='hoottest'" 
 
 # Same as above.
-hoot convert-osm2ogr \
+hoot convert \
   -D ogr.writer.pre.layer.name=bar \
-  test-files/io/SampleTranslation.js \
   test-files/io/SampleTranslation.osm \
-  PG:"dbname='osm_gis2' host='localhost' port='5432' user='hoot' password='hoottest'" 
+  PG:"dbname='osm_gis2' host='localhost' port='5432' user='hoot' password='hoottest'" --trans test-files/io/SampleTranslation.js
 
 # Convert example .osm file and write the resulting layers to a directory full of Shapefiles.
 # The directory will be named tmp/MyShapefiles, not tmp/MyShapefiles.shp

docs/commands/HootCommandLineReference.asciidoc

@@ -10,7 +10,7 @@ The base command line interface follows the following syntax:
 hoot (operation) [args...]
 -----
 
-Where +operation+ is an operation name such as +convert-ogr2osm+ or +conflate+. The
+Where +operation+ is an operation name such as +convert+ or +conflate+. The
 parenthesis around +operation+ show that the argument is required.
 
 +[args...]+ shows that there are zero or more arguments to the operation. The
@@ -26,16 +26,12 @@ include::accuracy-distribution.asciidoc[]
 include::attribute-co-occurrence.asciidoc[]
 include::attribute-count.asciidoc[]
 include::big-cat.asciidoc[]
+include::calculate-tiles.asciidoc[]
 include::changeset-apply.asciidoc[]
 include::changeset-derive.asciidoc[]
 include::clean.asciidoc[]
 include::conflate.asciidoc[]
-include::conflate-differential.asciidoc[]
-include::conflate-cumulative.asciidoc[]
 include::convert.asciidoc[]
-include::convert-ogr2osm.asciidoc[]
-include::convert-osm2ogr.asciidoc[]
-include::convert-osm2shp.asciidoc[]
 include::cookie-cut.asciidoc[]
 include::crop.asciidoc[]
 include::delete-db-map.asciidoc[]
@@ -70,8 +66,6 @@ include::score-matches.asciidoc[]
 include::sort.asciidoc[]
 include::stats.asciidoc[]
 include::tag-schema.asciidoc[]
-include::tiles-calculate.asciidoc[]
-include::tiles-calculate-random.asciidoc[]
 include::type-tagger-create-rules.asciidoc[]
 include::type-tagger-create-rules-db.asciidoc[]
 include::union-polygons.asciidoc[]

docs/commands/calculate-tiles.asciidoc

@@ -0,0 +1,60 @@
+== calculate-tiles 
+
+=== Description
+
+The +tiles-calculate+ command calculates a collection of bounding boxes that contain roughly equal distributions of node data from 
+the specified input(s) and writes the output to a file.
+
+* +input+               - one or more OSM data inputs; specify multiple inputs by separating them with a semicolon
+* +output+              - output file to write the tile boundaries to; valid formats are GeoJSON (*.geojson)
+* +maxNodeCountPerTile+ - The maximum number of nodes in any tile.  The default is 1000; optional only if pixelSize is not specified
+* +pixelSize+           - The size of the pixel used in the calculation in degrees.  The default is 0.001; optional only if 
+                          maxNodeCountPerTile is not specified
+* +random-seed+         - seeds the random number generator for consistent tile boundary selection; valid only if --random is specified; 
+                          use -1 for no seeding; defaults to -1
+* +--random+            - selects a single random tile from those calculated
+
+If too low a value is specified for maxNodeCountPerTile or too high a value is specified for pixelSize, the algorithm used may not be able 
+to find a solution.  
+
+This command makes use of the optional 'convert.bounding.box' configuration option to determine a subset of the input data to calculate 
+tiles for.  However, 'convert.bounding.box' is only currently supported for OSM API database or Hootenanny API database data sources.  
+If any other data source is used with the option, this command will return an error.
+
+=== Usage
+
+--------------------------------------
+hoot calculate-tiles (input(s)) (output) [maxNodeCountPerTile] [pixelSize] [random-seed] [--random]
+--------------------------------------
+
+=== Examples
+
+Calculate tiles based on node density:
+
+--------------------------------------
+hoot calculate-tiles test-files/MyInputFile.osm MyOutputFile.geojson
+--------------------------------------
+
+Calculate tiles based on node density, specifying the maximum allowed node count per tile and a pixel size:
+
+--------------------------------------
+hoot calculate-tiles "test-files/MyInputFile1.osm;test-files/MyInputFile2.osm" MyOutputFile.geojson 1000 0.001
+--------------------------------------
+
+Calculate tiles based on node density, narrowing down to a subset of the data specified by a bounding box:
+
+--------------------------------------
+hoot calculate-tiles -D convert.bounding.box="34.04,31.17,34.05,31.18" hootapidb://hoot:hoottest@localhost:5432/hoot/MyInputDb MyOutputFile.geojson
+--------------------------------------
+
+Select a random tile based on node density:
+
+--------------------------------------
+hoot calculate-tiles test-files/MyInputFile.osm MyOutputFile.geojson --random
+--------------------------------------
+
+Select a random tile based on node density specifying a particular seed for the random number generator:
+
+--------------------------------------
+hoot calculate-tiles test-files/MyInputFile.osm MyOutputFile.geojson 1000 0.001 2 --random
+--------------------------------------

docs/commands/conflate.asciidoc

@@ -3,26 +3,38 @@
 
 === Description
 
-The +conflate+ command conflates two maps into a single map.
+The +conflate+ command conflates two maps into a single map.  If the specified output file name ends with the extension .osc, an OSM 
+                     changeset will be written, instead of an OSM map.
 
-* +input1+  - First input.
-* +input2+  - Second input.
-* +output+  - The output path.
-* +--stats+ - Hootenanny map statistics such as node and way count
+* +input1+         - First input (e.g. OSM file).
+* +input2+         - Second input (e.g. OSM file, XML changeset file (.osc)).
+* +output+         - The output path.
+* +--differential+ - Calculates the the differential between two conflation inputs.  The output will be all elements in input2 that do 
+                     not conflict with any elements in input1.  
+* +--stats+        - Hootenanny map statistics such as node and way count
 
 === Usage
 
 --------------------------------------
-conflate (input1) [input2] (output) [--stats]
+conflate (input1) [input2] (output) [--differential] [--stats]
 ---------------------------------
 
-==== Example
+==== Examples
 
 --------------------------------------
 hoot conflate input1.osm input2.osm output.osm --stats
 --------------------------------------
 
+--------------------------------------
+hoot conflate input1.osm input2.osm output.osm --differential
+--------------------------------------
+
+--------------------------------------
+hoot conflate input1.osm input2.osm output.osc --differential
+--------------------------------------
+
 ==== See Also
 
 * _Unifying Conflation_, <<hootalgo,Hootenanny - Algorithms>>
+* _Differential Conflation_, <<hootalgo,Hootenanny - Algorithms>>