README.md

Bulk Load Features from ArcGIS Into Elasticsearch

Description

This ArcPy based tool uses the Python Elasticsearch Client API to bulk load the content of a feature class into Elasticsearch using the bulk REST API.

The tool retrieves the description of the fields in a specified feature class and creates an index mapping before the bulk import starts.

The geometry of point features can be stored as geo_points or geo_shapes. The geometry of polyline or polygon features is stored as an instance of geo_shape.

Each feature is converted it to a JSON document. In the case where the geometry of the feature is of type (multi)linestring or (multi)polygon, then it has to be converted to a GeoJSON element in the document.

Note: There is a little hidden gem in ArcPy, where you can get the GeoJSON representation of a geometry by accessing its __geo_interface__ property.

Here is a mapping snippet:

{
    "properties": {
        "shape": {
            "type": "geo_shape",
            "tree": "quadtree",
            "precision": "1km"
        }
    }
}

Here is an ArcPy snippet:

with arcpy.da.SearchCursor(fc, ['SHAPE@']) as cursor:
    for row in cursor:        
        geojson = row[0].__geo_interface__
        doc = {"shape":geojson}

There is a small "issue" in ES when it comes to polygons with multi parts, where the parts are siblings such as in the polygon.json file. Here is the view in geojsonlint.com that reports that the content of polygon.json is valid.

However, posting it using the bulk API:

curl -XPOST localhost:9200/_bulk --data-binary @request

Results in the following error:

{
	"took": 4,
	"errors": true,
	"items": [{
		"index": {
			"_index": "miami",
			"_type": "extent",
			"_id": "1000",
			"status": 400,
			"error": "MapperParsingException[failed to parse [shape]]; nested: InvalidShapeException[Invalid shape: Hole is not within polygon]; "
		}
	}]
}

To get around this, there is an option in the tool to convert Polygons to MultiPolygons. Make sure to check that on !

Trying The Tool

Download the sample AIS data in FileGeodabase format. It contains a point feature class of 1.3 million ship locations with timestamps, headings, ship identifiers (MMSI) and more attributes.

Starting Elasticsearch.

We will use Docker to create a local instance of Elasticsearch using the new Docker Toolbox.

docker-machine create --driver virtualbox --virtualbox-memory "4096" --virtualbox-cpu-count "4" dev
export DEV=$(docker-machine ip dev)
eval "$(docker-machine env dev)"

Start an ES instance based on developer centric configuration files in the config folder.

docker run -d -p 9200:9200 -p 9300:9300 -h dev -v "$PWD/config":/usr/share/elasticsearch/config elasticsearch

Note: The dev value to the -h option defines the container host name - this is hardcoded in the config/elasticsearch.yml for easy discovery, development and testing.

Check if ES is up and running:

open http://$DEV:9200

This should result in something like the following:

{
	status: 200,
	name: "Akhenaten",
	cluster_name: "elasticsearch",
	version: {
		number: "1.7.1",
		build_hash: "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
		build_timestamp: "2015-07-29T09:54:16Z",
		build_snapshot: false,
		lucene_version: "4.10.4"
	},
	tagline: "You Know, for Search"
}

Invoking the Bulk Loader Tool

Update your PATH system environment variable to include Python. In my case it looks like:

C:\Python27\ArcGIS10.3;C:\Python27\ArcGIS10.3\Scripts

Then use pip to install elasticsearch.

pip install elasticsearch

Start ArcMap and add from miami.fgb the Broadcast feature class. Open the Bulk Load Features tool in the ElasticsearchToolbox.pyt toolbox.

The ES Host(s) value is the return of the docker-machine ip dev command on the host, and make sure to check the Index Points as Geo Shapes for this walk through.

Note: This process takes about 20 minutes on my MacBook Pro with ArcMap running in WMWare Fusion and ES in VirtualBox, so...go and grad a cup of coffee or clean your office :-) This is not the most efficient way to bulk load features into ES. One of these days will have to come back and rewrite this tool to take advantage of massive parallel reading.

Using Sense in Chrome (You are using Chrome, right? I mean, what else is there?), perform the following query:

Invoking the Geo Search Tool

I like ES because it can store and spatially query spatial types. The GeoSearchTool relies on the Geo Distance Filter to query all documents that are within a radius distance from a specified location.

Note that the Type Name property is optional, as the tool will query the whole index if the Type Name value is blank. The field mapping property enables the user to define what properties to retrieve from the queried documents. Some properties might be included in some documents and others will not be. That is the beauty of document stores. However, ArcGIS FeatureClasses still have a row/column model and non-existing document properties will be populated as null values in the feature attributes.

TODO: On Index Name and Type Name parameter updates, query ES for the mapping and auto populate the Fields parameter.

Final Notes

You can gracefully shutdown ES using:

curl -XPOST $(docker-machine ip dev):9200/_cluster/nodes/_shutdown

An ES node publishes its IP to communicate with it. However in this case, it publishes it docker container IP typically in the form 127.17.0.XXX. So on my mac, I had to reroute the traffic when communicating with it using:

sudo route add -net 172.17.0.0/16 $(docker-machine ip dev)

You can get the internal docker IP using the following to get the list of the containers:

docker --tls ps

Find the active CONTAINER_ID from the list, and replace it in the following command:

docker --tls inspect -f=“{{.NetworkSettings.IPAddress}}” CONTAINER_ID