Services cannot be consumed in other Esri clients (Pro/Earth)

[WalterPayne]

My service works well in Portal for ArcGIS as I can create a web map containing my service.

Unfortunately I cannot consume the service in ArcGIS Earth.
I get the following error:
"expecting value type to be value_string":

Since KOOP isn't a full blown ArcGIS server, in Pro you cannot create an ArcGIS Server connection.
I did create a web map in Portal and tried to add it to Pro.
It only showed the basemap and not my feature layer

I then created a Portal item that contains the link to my service and tried adding that to Pro. The error was: Cannot add data.

Any plans to get the feeds working in other clients?

Thanks again,

Walter

[jgravois]

using a tool like fiddler, what do the request(s) and response(s) to the koop service look like?

back in the day ArcGIS desktop used SOAP to talk to web services...

[WalterPayne]

Feel free to have a look. ArcGIS Earth can be downloaded here:
http://www.esri.com/software/arcgis-earth

I strongly doubt Pro or Earth are using SOAP. There may be a formatting issue in the KOOP responses that it's struggling to parse.

[rminderhoud]

I agree with @WalterPayne. Would be nice for easier integration with ArcGIS products that expect a full ArcGIS server for a connection (e.g. ArcCatalog, ArcMap, ArcScene, etc.). I haven't found a way yet, but maybe there's a way to couple a koop instance and an ArcGIS server?

[jgravois]

we're currently investigating the following error in ArcGIS Pro

POST
f=json&
outFields=OBJECTID&
where=1=1&
resultOffset=0&
resultRecordCount=1000&
returnGeometry=true&
maxAllowableOffset=0.00071641681087762643&
outSR={"wkid":4326,"latestWkid":4326}&
geometry{"xmin":-77.7681498738757426,"ymin":38.5091641988065021,"xmax":-76.6569265444183543,"ymax":38.8867767139252365,"spatialReference":{"wkid":4326,"latestWkid":4326}}&
geometryType=esriGeometryEnvelope&
spatialRel=esriSpatialRelIntersects&
orderByFields= OBJECTID ASC

returns

{"error":"Parse error on line 1:\n... BY properties->`` OBJECTID LIMIT 1000\n-----------------------^\nExpecting 'EOF', 'WITH', 'COMMA', 'RPAR', 'IN', 'LIKE', 'ARROW', 'DOT', 'CARET', 'EQ', 'SLASH', 'EXCLAMATION', 'MODULO', 'GT', 'LT', 'GTGT', 'LTLT', 'NOT', 'UNION', 'INTERSECT', 'EXCEPT', 'AND', 'OR', 'PLUS', 'STAR', 'DIRECTION', 'COLLATE', 'LIMIT', 'OFFSET', 'END', 'ELSE', 'REGEXP', 'TILDA', 'GLOB', 'NOT_LIKE', 'BARBAR', 'MINUS', 'AMPERSAND', 'BAR', 'GE', 'LE', 'EQEQ', 'EQEQEQ', 'NE', 'NEEQEQ', 'NEEQEQEQ', 'BETWEEN', 'NOT_BETWEEN', 'IS', 'DOUBLECOLON', 'GO', 'SEMICOLON', got 'LITERAL'"}

[jptheripper]

I have just tried a feature service in pro 2.1. It does not work it displays fine in a portal mapping application, and can be identified, but the attribute table doesnt load). When it is brought into Pro via add data from path it shows up in the table of contents. It does not draw. Loading the attribute table throws

---

Table

Failed to load data.

Error:
A networking error occurred.
Internal server error. (status code 500).

_OK

For some reason it wont let me attach the fiddler log, so its here

http://www.mymanatee.org/gisapps/data/other/pace_koop_arcgispro.zip

[rgwozdz]

@jptheripper - I am attempting to reproduce the behavior you have noted. I am able to bring a Koop service into Pro. Initially, it does appear in the table of contents, but is not rendered on the map. However, if I right-click on the layer and choose "Zoom to layer", it does then display on the map view. Can you confirm this works for you as well?

I find the same error on attempting to view the attribute table.

[rgwozdz]

@WalterPayne - In Pro, are you trying to add a Koop service with the "Data From Path" control?

[WalterPayne]

When you say "I'm able to bring a Koop service into Pro", how are you doing that?

My method was to add my Koop service as a Portal item first, then add it to Pro.

I suspect that when the attribute table error is resolved it may solve some other connectivity issues as well.

[rgwozdz]

@WalterPayne - the way I am testing this is to deploy a Koop instance that exposes some service URLs. Then, in ArcGIS Pro, I add that service with the "Data from Path" control. So I am leaving out the Portal step. Thank you for the clarification.

[rgwozdz]

@jgravois - After stepping through the code with the request parameters you posted, it appears that parse error you noted above is due to the leading whitespace on the value of the orderByFields parameter. We might consider adding a whitespace trim to the middleware, or kicking back a Bad Request error if any of the parameter values contain leading/trailing whitespace.

However, removal of that whitespace still doesn't solve the failure to load the attribute table noted by @jptheripper and @WalterPayne.

[jptheripper]

OK here is what i found

1. I got koop services to work in WAB. I had to override the returncountonly method to handle all the different requests of the attribute table
2. Koop services do not work in Pro. They load, i can see the legend, but the first request Pro makes is a POST, not a GET. Which fails of course, since I believe the koop providers only handle GETs.

[jptheripper]

Looking at koopjs/koop-provider-agol#16 Posts are supported, however when i post it fails. the exact same request as a get is successful. I will have to troubleshoot

…

On Wed, Mar 21, 2018 at 2:50 PM, Rich Gwozdz ***@***.***> wrote: @jgravois <https://github.com/jgravois> - After stepping through the code with the request parameters you posted, it appears that parse error you noted above is due to the leading whitespace on the value of the orderByFields parameter. We might consider adding a whitespace trim to the middleware, or kicking back a Bad Request error if any of the parameter values contain leading/trailing whitespace. However, removal of that whitespace still doesn't solve the failure to load the attribute table noted by @jptheripper <https://github.com/jptheripper> and @WalterPayne <https://github.com/walterpayne>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#304 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKEroM1zTkTyiffdYZkQA7DKTI6HFXdwks5tgqERgaJpZM4Pv97R> .

[jptheripper]

Finally got the error

{
"error": "Parse error on line 1:\n... BY properties->`` OBJECTID LIMIT 50000\n-----------------------^\nExpecting 'EOF', 'WITH', 'COMMA', 'RPAR', 'IN', 'LIKE', 'ARROW', 'DOT', 'CARET', 'EQ', 'SLASH', 'EXCLAMATION', 'MODULO', 'GT', 'LT', 'GTGT', 'LTLT', 'NOT', 'UNION', 'INTERSECT', 'EXCEPT', 'AND', 'OR', 'PLUS', 'STAR', 'DIRECTION', 'COLLATE', 'LIMIT', 'OFFSET', 'END', 'ELSE', 'REGEXP', 'TILDA', 'GLOB', 'NOT_LIKE', 'BARBAR', 'MINUS', 'AMPERSAND', 'BAR', 'GE', 'LE', 'EQEQ', 'EQEQEQ', 'NE', 'NEEQEQ', 'NEEQEQEQ', 'BETWEEN', 'NOT_BETWEEN', 'IS', 'DOUBLECOLON', 'GO', 'SEMICOLON', got 'LITERAL'"
}

I believe this may be related to pagination but i am guessing wildly

[jptheripper]

well, i got around this error by removing the orderbyfields parameter as well, and got a point or two to display in Pro, but the renderer was wrong, attriburtes are blank, and there should have been alot more points. My guess is lack of objectid in source data, even though i spoof one.

[jptheripper]

"failed to retrieve a page of rows"
could these problems be related to the Koop server saying pagnation is supported even though it isnt?

[rgwozdz]

@jptheripper - Thanks for your comment. I've been diving deep into this and there are a number of different issues contributing. But as you noted the parse error appears to be caused by the client (ArcGIS Pro, in my case) adding some leading white space to the request parameter orderByFields, e.g., orderByFields=%20OBJECTID%20ASC. We released a fix for this in koop-core v3.5.4 on Friday.

[rgwozdz]

It appears there may be a few different issues at work in the errors reported in this thread. For clarity, I have separated them into different issues for more granular tracking purposes (see above) and included you all.

[jptheripper]

I believe I am getting closer. Is there a way to change a koop featureserver from 4326 to 3857 as the default spatial reference? I think that is part of my problem

[rgwozdz]

@jptheripper - I think I have this figured out, at least for Pro. Part of the problem for loading the attributed table was caused by geometry being returned in response body despite the returnGeometry option being explicitly set to false. Fixed in koopjs/winnow@dfffbe2 and just bumped/published winnow.

I have a separate PR (koopjs/winnow#62) in that fixes the rendering issue noted here #317.

However, there is a blocker for ArcGIS Online Scene Viewer and possibly ArcGIS Earth (#318).

[rgwozdz]

TLDR: With FeatureServer v2.12.1 and winnow v1.15.2, Koop services can be consumed by ArcGIS Pro - table loads, features render, popup works.

Details:
Pro functionality is sensitive to data-type and values of OBJECTIDs. They need to be integers with range of 0 - 2,147,483,647. If you are setting idField in your model metadata to a non- or out-of-range integer, Koop services will likely not work properly with Pro. Warnings have been added to FeatureServer when a model doesn't not have a idField or it is the wrong data type.

The latest updates of winnow(1.14.0) also changes the approach for creating OBJECTIDs in the event idField was not set. If creating ESRI JSON, winnow generates the OBJECTID from the numeric-hash of the feature. This doesn't completely eliminate the chance of an OBJECTID collision, so you are better off if you can set idField in your model to a unique, in-range, integer.

Inability to consume Koop services with ArcGIS Earth and SceneViewer remain. Will continue those as separate issues.

[rgwozdz]

@WalterPayne - if you get the chance and are still interested, would you try upgrading FeatureServer and winnow and then use your Koop with Pro? We have tested with Scene Viewer and Pro and are having success. Earth is still problematic - partially solved by using beta 1.7, but still working through downstream rendering issues.

[rgwozdz]

Closing as we have tested Pro with the Craigslist provider, FeatureServer v2.12.1 and winnow v1.15.2 Success with loading, rendering, attribute table loading, identify and zoom-to.
Koop -> ArcGIS Earth not yet supported; moving that here #323.

[jptheripper]

Unfortunately this still isnt working for me and i have to use the workaround of making my own objectid. should i make a new incident?

[rgwozdz]

My gut feeling is that this is something specific to your provider, but perhaps you have something that our Craigslist provider doesn't trigger. Yes, please create a new issue - this issue was fairly general ("services can't be consumed") and as a result it could stay open for a long time (users are likely to have this issue for a variety of reasons). If you could open a more detailed issue that would be great. Also if you could link to your provider repository (if it's open source), that would be helpful - as I can't reproduce the issue with other providers.