Adds ESRI map service support

[eliotjordan]

This pull request adds support for ArcGIS REST API services. @cheetah90 from UMN kicked off the effort and did the preliminary work. This is a rough draft and really needs more eyeballs on it from the community.

Missing Features:

• Feature layer inspection
• Web services link
• Download link (punting in this PR)

Areas for Closer Inspection:

• Reference URI (Currently: http://resources.arcgis.com/en/help/arcgis-rest-api)
• Fixtures

[coveralls]

Coverage increased (+0.02%) to 95.18% when pulling 1e569f6 on cheetah90:esri-map-viewer into 9cd70e4 on geoblacklight:master.

[mejackreed]

So I haven't been able to dive into this fully yet but a quick observation:

We are referencing both a "MapServer" and "FeatureServer" endpoint with the same URI. These seem like different enough separate services, no?

[eliotjordan]

In that case, it would be three: "MapServer", "FeatureServer", and "ImageServer". That was my original thought. Digging in, however, the reality is quite a bit messier. For instance, a map service can be a tiled map cache, a dynamic layer, or can even posses a number of constituent layers that might actually be feature layers ('../MapServer/0'). A map service that behaves a tiled map cache is often what we think of a single 'layer'. In other instances, a map service is more like a folder/grouping of individual layers.

A feature service generally behaves as a folder, so what you are really after is the individual feature layer ('../FeatureServer/0').

The ArcGIS 'Geo Services' REST API is a hulking and fluid thing that changes somewhat with each release (yearly). But you're right, we could, and possibly should, have a separate reference for each service type. I was just trying to keep things simple. Poke around and see what makes sense here.

Possible Ref URIs:

http://resources.arcgis.com/en/help/arcgis-rest-api/#/Map_Service
http://resources.arcgis.com/en/help/arcgis-rest-api/#/Feature_Service
http://resources.arcgis.com/en/help/arcgis-rest-api/#/Image_Service

[eliotjordan]

For clarity and reference, these are the layer types that we are interested in by service type.

Map Service

• Tiled Map Layer
  • URL: '../MapServer'
  • Can be inferred from service metadata
  • singleFusedMapCache === true
• Dynamic Map Layer
  • URL: '../MapServer'
  • Can be inferred from service metadata
  • singleFusedMapCache === false
  • supportsDynamicLayers !== false # often dynamic layers do not have this property (yay!)
• Feature Layer
  • URL: '../MapServer/{id}'
  • Can be inferred from url and then type checked on layer metadata
  • type === 'Feature Layer'

Feature Service

• Feature Layer
  • URL: '../FeatureServer/{id}'
  • Can be inferred from url and then type checked on layer metadata
  • type === 'Feature Layer'

Image Service

• Image Map Layer
  • URL: '../ImageServer'
  • Can be inferred from url

[mejackreed]

Great comment @eliotjordan ! 👏 Let us digest this a bit and we will follow up.

Thanks @eliotjordan and @cheetah90 for the excellent work!!

[drh-stanford]

Since there's so much logic in determining the actual web services, I'd suggest a "meta" Esri service URL that you put in dct_references_s. The URL would point to a parent path to all those possible endpoints. Unfortunately, it means additional logic in the application to generate the correct endpoints, but that looks unavoidable from your above comment. I read that as any layer can have one or more valid endpoints but all layers do not have all three.

[eliotjordan]

@drh-stanford Do you mean that a layer's dct_ref would be something like this?

{"http://resources.arcgis.com/en/help/arcgis-rest-api": "http://gis.com/arcgis/rest/services/"}

From what I can tell, that might work in some cases, but, in general, you really need to be explicit as to where to find the thing you are looking for. I'll put together a few examples.

[eliotjordan]

( append ?f=pjson to get pretty JSON )

This is an example of where a 'meta' reference might work:

layer_id: USGSTopoLarge
dct_refs: {"http://resources.arcgis.com/en/help/arcgis-rest-api":"http://services.nationalmap.gov/arcgis/rest/services"}

In the services metadata you find:

{
    name: "USGSTopoLarge",
    type: "MapServer"
}

And now you can determine if the layer is a tile map or a dynamic map.

This is a working example with a feature service:

layer_id: Buildings
dct_refs: {"http://resources.arcgis.com/en/help/arcgis-rest-api":"http://services.arcgis.com/afSMGVsC7QlRK1kZ/ArcGIS/rest/services"}

In the services metadata you find:

{
    name: "Buildings",
    type: "FeatureServer",
    url: "http://services.arcgis.com/afSMGVsC7QlRK1kZ/ArcGIS/rest/services/Buildings/FeatureServer"
}

And if you get the layers in the FeatureServer

{
    id: 0,
    name: "Buildings",
    type: "Feature Layer",
    ...
}

On the other hand... on the same server:

dct_refs: {"http://resources.arcgis.com/en/help/arcgis-rest-api":"http://services.arcgis.com/afSMGVsC7QlRK1kZ/ArcGIS/rest/services"}

You are looking for a solar suitable buildings layer. Which ID do you use?

layer_id: BuildingsSolarSuitability

In the services metadata you find:

{
    name: "BuildingsSolarSuitability",
    type: "FeatureServer",
    url: "http://services.arcgis.com/afSMGVsC7QlRK1kZ/ArcGIS/rest/services/BuildingsSolarSuitability/FeatureServer"
}

In the feature server layers we find two layers with different names:

{
    id: 0,
    name: "Solar Suitable Buildings",
    ...
    id: 1,
    name: "Buildings Not Suitable",
} 

Also, you might want a layer within a MapService (very common):

layer_id: 'MetroGIS Parcels'
dct_refs: {"http://resources.arcgis.com/en/help/arcgis-rest-api":"http://gis.hennepin.us/arcgis/rest/services/"}

Looking at the services metadata only gives an array of folders. In these folders we eventually find our layer in the "HennepinData/LAND_PROPERTY" service.

http://gis.hennepin.us/arcgis/rest/services/HennepinData/LAND_PROPERTY/MapServer/3

[drh-stanford]

Thanks for the examples. It looks to me that they have a notion of a nested group of layers. That is BuildingsSolarSuitability is actually a "container" consisting of 2 layers (id:0 and id:1). So wouldn't you have metadata records for each where the layer_id is BuildingsSolarSuitability/0 and BuildingsSolarSuitability/1?

Same thing with the second example, HennepinData/LAND_PROPERTY would be its own metadata record, distinct from "MetroGIS Parcels" which is actually a collection of layers.

Am I understanding your example correctly? I'm looking at the figure here: http://resources.arcgis.com/en/help/arcgis-rest-api/#/Layer/02r3000000w6000000/ and in their terminology, our layer would be a Feature Layer. But I'm not sure I understand exactly why their feature server (service?) layers are different from just feature layers.

[eliotjordan]

@drh-stanford Yeah. This makes my head hurt.

FeatureServer services contain Feature Layers 😵. But MapServer services can also contain Feature Layers. On their own, FeatureServer services aren't 'layers' in the way that we want to use them. However a MapServer service, on its own, might be 'layer' in that way (a tiled map of soil data), but it might not be.

Assigning the geoblacklight item an identifier such as HennepinData/LAND_PROPERTY/MapServer/3 is definitely a solution. Kinda gnarly, but workable.

[cheetah90]

@eliotjordan thanks for pushing it forwards so much! Just coming back from a graduate vacation and trying to catch up here.

I think the whole confusion around "Layer" vs. "Server" vs. "Service" might be due the differences between ArcGIS Server and ArcGIS Online.

In ArcGIS Server terminology http://resources.arcgis.com/en/help/main/10.1/index.html#//0154000002w8000000, Map/Feature/Image Service" (or Server, as you see in the metadata) can contains multiple layers, indexed by either layer name or a numerical id. (When I was working with in OGP, the layer name does not work with OpenLayer 2's API so I have to use the numerical id. Not sure if that's the case for Esri-Leaflet plugin we used) The URL endpoint points to the server/service, not to the specific layer.

ArcGIS Online can also web hosts "layers" https://doc.arcgis.com/en/arcgis-online/share-maps/hosted-web-layers.htm. The URL of these layers are started with "services.arcgis.com". They are called Feature Layer but in my understanding, they are simply the "layers" in the FeatureServer.

As per another thread, @mejackreed what example you want me to give?

BTW, I am currently in Redlands doing a summer internship in Esri's Javascript API team. Will be happy to coordinate the communication between GeoBlacklight community and Esri if needed.

[eliotjordan]

Hey all! I've set aside some time this week to push the ESRI module forward. The dct_references issue needs resolution before we can continue. Any thoughts or strong opinions (old and new) on this? After looking over the comments, it seems to me that the simplest solution is to do what @drh-stanford suggested, and define a single rest-api reference for the server and then use the item's layer_id to find the specific layer. This mirrors the approach we take with WMS and WFS services.

layer_id: 'HennepinData/LAND_PROPERTY/MapServer/3',
dct_refs: {"http://resources.arcgis.com/en/help/arcgis-rest-api":"http://gis.hennepin.us/arcgis/rest/services"}

The client can determine the appropriate esri layer type from the metadata and layer_id. Anyone else?
@cheetah90 @mejackreed @gravesm

[gravesm]

Yeah, this seems like a reasonable approach.

[mejackreed]

I'm a little apprehensive about using an additional field to store (I'm also apprehensive about how we do it currently for WMS/WFS).

The approach would not allow for a layer to have both services in ESRI and WMS/WFS format. I'm not sure if this is a use case for some, but would hate to disable this functionality.

I'm leaning more and more to explicitly providing the type of service:

{
  "http://resources.arcgis.com/en/help/arcgis-rest-api#mapserver":"http://gis.hennepin.us/arcgis/rest/services/HennepinData/LAND_PROPERTY/MapServer/3"
}

Also happy to step aside and proceed as the group feels best if what I'm saying doesn't make sense.

[eliotjordan]

@mejackreed You make a good point about losing the ability to have WMS services and REST services simultaneously. I too have wondered why the layer_id should be a separate field and why not just explicitly define it in the dct_ref. It certainly allows for more flexibility. And I like the directness of it.

Its clear to me that the the various 'services' are probably the most straightforward logical units here. They each posses unique functionality, even if there is overlap among them. I'm OK with three ref URIs - Map_Service, Feature_Service, and Image_Service.

[eliotjordan]

Another question/concern. Given ESRI's habit of overhauling their documentation periodically, can we reasonably expect that 'http://resources.arcgis.com/en/help/arcgis-rest-api' will resolve for the foreseeable future? I don't have any real alternatives in mind, so maybe this a moot point.

[mejackreed]

Yeah @eliotjordan 👍 on the docs stability question??? (ツ)/¯

Back to your email a few weeks ago about this:

• http://www.arcgis.com/rdf/Map-Service
• http://www.arcgis.com/rdf/Feature-Service
• http://www.arcgis.com/rdf/Image-Service

What about something like this?:

• http://www.arcgis.com/rest/services/MapService
• http://www.arcgis.com/rest/services/FeatureService
• http://www.arcgis.com/rest/services/ImageService

[drh-stanford]

It would be nice to support a GeoBlacklight layer having all protocols enabled. This is where the layer_id in the schema breaks down because it's possible that the layer id actually varies between the dct_references protocols. It looks like ArcGIS forces this point.

I'm not sure how to solve this moving forward without breaking backward compatibility, other than leveraging JSON-LD's triples. (Unfortunately) the JSON-LD way of solving this would be something like this:

"http://resources.arcgis.com/en/help/arcgis-rest-api#Endpoint":
"http://gis.hennepin.us/arcgis/rest/services"

"http://resources.arcgis.com/en/help/arcgis-rest-api#MapServer-FeatureID":
"HennepinData/LAND_PROPERTY"

"http://resources.arcgis.com/en/help/arcgis-rest-api#MapServer-FeatureLayerID":
"3"

Where the application constructs the URL using the predicate-values:

URL = endpoint + '/' + mapServerFeatureID + '/MapServer/' + mapServerFeatureLayerID

(Note that I'm still confused about their protocol, but that's the idea)

Moving forward, this would work for WxS too by putting the actual layer id as its own triple and even a URL template for constructing it (e.g., wms#url-template="...wms?layer=#{layerId}..." and "...wms#layerId" = foobar) and deprecating layer_id. That's clearly a 2.0 type feature but the triple way would work.

As for the URI, the general rule is to use the protocol's URI if provided, otherwise point to the specification, otherwise make one up. It looks like CatInterop made one up (urn:x-esri:serviceType:ArcGIS). We could adopt that or make up our own if we're leary about using the specification pointer.

[mejackreed]

The JSON-LD approach is interesting... I'm curious how others would be able to approach this, or what added benefits this might give us?

One thing I was thinking with WxS was just moving to the specified parameter in the spec:

{
// WMS use "layers" param for getMap requests
  "http://www.opengis.net/def/serviceType/ogc/wms" : "http://www.example.com/geoserver/wms?layers=layerid_123"

// WFS use "typeNames" param for getFeature requests
  "http://www.opengis.net/def/serviceType/ogc/wfs" : "http://www.example.com/geoserver/wfs?typeNames=layerid_123"
}

This does seem to duplicate the same information though.

[drh-stanford]

For this example, the JSON-LD approach would be to pass the parameters in as their own predicate-values, like so:

dct_references_s = {
  # As-is today
  "http://www.opengis.net/def/serviceType/ogc/wms" : "http://www.example.com/geoserver/wms",
  # use this rather than layer_id in schema
  "http://www.opengis.net/def/serviceType/ogc/wms#layerId" : "layerid_123",

  # As-is today
  "http://www.opengis.net/def/serviceType/ogc/wfs" : "http://www.example.com/geoserver/wfs",
  # use this rather than layer_id in schema
  "http://www.opengis.net/def/serviceType/ogc/wfs#layerId" : "layerid_987" # note difference in value
}

Alternatively, if you wanted to embed the application logic for how to construct the URL you could use a template like so:

{
  "http://www.opengis.net/def/serviceType/ogc/wms#template" : "http://www.example.com/geoserver/wms?layers=#{layerId}",
  "http://www.opengis.net/def/serviceType/ogc/wms#layerId" : "layerid_123",

  "http://www.opengis.net/def/serviceType/ogc/wfs#template" : "http://www.example.com/geoserver/wfs?typeNames=#{layerId}",
  "http://www.opengis.net/def/serviceType/ogc/wfs#layerId" : "layerid_987" # note difference in value
}

The benefit of this approach is that all the protocol-specific parameters are encoded in dct_references_s rather than as layer_* parameters in the schema. The layer_id parameter was meant to be the same for all protocols, but it's not always as ArcGIS demonstrates.

[eliotjordan]

I'm also interested in JSON-LD and wonder what the benefits are to expressing dct_references in this way. These are certainly breaking changes. So... do we need to do this work now, before we proceed to adding new functionality (ESRI Layers)? Or do we wait until a 2.0 release as @drh-stanford mentioned? We are not yet at 1.0, but there is at least one significant instance already in production.

[mejackreed]

I like the JSON-LD approach... I feel like the templating could start to get too complicated but providing the layer_id seems easy enough. I'm pretty sure this could be done in a backward compatible way.

One thing I want to be cautious of here is making things too complicated. That may hinder others to use/adopt/contribute to the software. The simplest solution may win here in the long run.... I've been looking into how data.gov and ckan work with this, I'm not sure what the solution is yet.

[eliotjordan]

Perhaps we should come up with a few examples of dct_references in json-ld. I'm toying with it now, and it seems easy to get complex very quickly.

Another thought. If we use JSON-LD for dct_references, why wouldn't we use JSON-LD for the entire document? Complexity for the end user? Issues with Solr?

[drh-stanford]

dct_references_s is technically a JSON-LD object as-is today (but it's implemented as a Solr string). It's just a trivial one. That's where the URI -> value structure comes from -- we just don't use the context stuff and we imply the subject. We could introduce the context but I'd rather use fully-qualified URIs otherwise it gets complex quickly.

As for using JSON-LD for the whole thing, I don't see a technical reason we couldn't do that. The main problem is getting metadata out of ISO/FGDC. If we had really good metadata extraction tools for that then new adopters wouldn't necesarily see or care about a JSON-LD implementation. I'm not really sure exactly what problem that solves though, other than consistency, or maybe interoperability with other open data platforms?

[eliotjordan]

👍 on meetup. Sunday is good for me. Flight gets in at noon, so anytime after 3 or 4?

[ajturner]

I don't land until very late Sunday night. So how about Monday evening?

[mejackreed]

@ajturner Monday works.

[ajturner]

Anyone that wants to meet up - can you email me your address? I'm andrew@...

[eliotjordan]

I sketched out two possible paths for moving forward. The first creates three reference types in dct_refs and separate viewer modules for each.

https://github.com/cheetah90/geoblacklight/pull/2/files

The second approach has one reference type in dct_refs and differentiates between the services via the URI fragments (#mapService etc...). The single viewer module parses the fragment.

https://github.com/cheetah90/geoblacklight/pull/3/files

Thoughts? How should we proceed?

[drh-stanford]

👍 on the fragment (PR 3) approach

[mejackreed]

Sorry here... I may of mixed up the two approaches. I'm 👍 for https://github.com/cheetah90/geoblacklight/pull/2/files (the first approach) named "Multi ref #2". Creating the multiple references feels more maintainable than extracting out the reference fragments.

[gravesm]

👍 for PR 2

[eliotjordan]

Hey everyone. Due to some airline awesomeness, I've been stuck at JFK airport for the last 8 hours. On the plus side, I've been hacking away at this PR. I've come to the conclusion, as I swim deeper and the water gets murkier, that we should really not be using ESRI services (map service, feature service) as the logical units for ESRI endpoints. What we should be doing is defining them as 'layers' even though that definition is a bit aspirational. The ArcGIS API for Javascript is really useful here. What follows is another proposal/sketch. I hope that we can hash out some of URI issues on Monday at the Linked Data Drinkup. When I (eventually) land in SD, I'll push up the code that I have.

---

In the ArcGIS universe, layers can be single datasets (e.g. single tables, feature layers), pre-rendered composites of one or more layers (tiled map service), dynamically rendered composites of one or more layers (dynamic map service), or imagery services. There are others, but these seems the most relevant.

FeatureLayer

• From Tiled Map Service: http://server.arcgisonline.com/ArcGIS/rest/services/NGS_Topo_US_2D/MapServer/0
• From Dynamic Map Service: http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Louisville/LOJIC_LandRecords_Louisville/MapServer/0
• From Feature Service: https://services.arcgis.com/rOo16HdIMeOBI4Mb/arcgis/rest/services/Neighborhoods_pdx/FeatureServer/0

ArcGISTiledMapServiceLayer

• http://server.arcgisonline.com/ArcGIS/rest/services/NGS_Topo_US_2D/MapServer

ArcGISDynamicMapServiceLayer

• Specify layer: http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Louisville/LOJIC_LandRecords_Louisville/MapServer/0
• Display all: http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Louisville/LOJIC_LandRecords_Louisville/MapServer

ArcGISImageServiceLayer

• http://sampleserver6.arcgisonline.com/arcgis/rest/services/Toronto/ImageServer

With something like these as URIs:

http://www.arcgis.com/rdf#FeatureLayer
http://www.arcgis.com/rdf#ArcGISTiledMapServiceLayer
http://www.arcgis.com/rdf#ArcGISDynamicMapServiceLayer
http://www.arcgis.com/rdf#ArcGISImageServiceLayer

[mejackreed]

@eliotjordan Holler when you get here. Since we are using the esri-leaflet library, can we just make the distinction using the API they already establish?

http://esri.github.io/esri-leaflet/api-reference/services/service.html

• L.esri.Layers.DynamicMapLayer
• L.esri.Layers.ImageMapLayer
• L.esri.Layers.RasterLayer
• L.esri.Layers.TiledMapLayer
• L.esri.Layers.FeatureLayer

[eliotjordan]

Pushed new commit that follows the multiple layer/service strategy.

[mejackreed]

@eliotjordan Looks good, I'm reviewing now. Can we change the cursor for the inspectable layers?

https://github.com/geoblacklight/geoblacklight/blob/master/app/assets/stylesheets/geoblacklight/modules/item.scss#L5-L7

Looks like this is also broken for WMS layers as well. We probably should have more intelligent css selectors.

[mejackreed]

I think this line is missing after LN62

$('.attribute-table-body').html(spinner);

[mejackreed]

We are including an esri-leaflet version that is v1.0.0-rc.7. Can we include the latest stable version 1.0.0 instead? https://github.com/Esri/esri-leaflet/releases/tag/v1.0.0

[mejackreed]

Can we also test here that the layer opacity is being setup? There seems to be a bug where the layer opacity indicator is started at 75% but the text is saying 100%

FeatureLayer opacity doesn't seem to be working correctly

[eliotjordan]

Made corrections and additions. Opacity control is now working correctly as well as cursor changes during inspection. Feature layer was particularly difficult because of its use of svg elements.

[eliotjordan]

I am closing this PR in order restart this discussion and to facilitate updating and moving this code into a branch on the main repository.