Errors when clicking on the lower Mississippi

[JamesBisese]

We're still getting errors when clicking on the lower Mississippi. the attached graphic TT_error.png shows the error and a run-away request from the server. The request when I grabbed the screen capture there had been nearly 10k requests and it was still going.

[JamesBisese]

This error is occurring because of deficiencies in the setup and configuration of the ESRI ArcGIS Server enviroatlas.epa.gov, and specifically the 'subwatershed' Map Service
https://enviroatlas.epa.gov/arcgis/rest/services/Other/HydrologicUnits/MapServer/4/query

To Recreate:

1. Edit code to use the 'enviroatlas.epa.gov' ArcGIS Server instead of the pre-configured 'watersgeo.epa.gov' where these errors do not occur.

2. Note: These tests are being done using a development server that is using HTTP and not HTTPS. Browsers (ie. Chrome) has a setting to reduce the number of async queries that can be active using HTTP to 6. This throttling does not happen if the application is run using HTTPS.

3. open a web browser, and open the browsers developer tools

4. navigate upstream from a single HUC12 by clicking on the map. I suggest Little Rock Arkansas, an d specifically 111102070402 Faulkner Lake - Arkansas River

5. this HUC12 has 4,088 upstream HUCs and the area upstream is 404,240 km2

6. the developer tools console will display messages about the status of the navigation and the GIS queries

   ++++ user clicked map. running initial HUC12 query ++++
   index.js:524 initial HUC12 query finished: [{…}]0: displayFieldName: "HU_4_Name"features: [d]fieldAliases: {OBJECTID: "OBJECTID_1", HUC_12: "HUC_12", Pct_Land: "Pct_Land", HUC4: "HUC4", HUC8: "HUC8", …}fields: (8) [{…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}]geometryType: "esriGeometryPolygon"spatialReference: f {wkid: 102100, latestWkid: 3857}transform: null_ssl: undefined__proto__: Objectlength: 1__proto__: Array(0)
   index.js:537 initial HUC12 query succeeded and found 1 features
   index.js:879 ++ wbd navigation finished in 56095.090ms
   index.js:958 there are 4089 HUC12s and 103 HUC8s upstream
   index.js:995 running 46 HUC12 GIS queries
   index.js:1035 ++ gis queries started. 46 HUC12 and 1 HUC8 upstream queries

7. The navigation query is done in 56,095ms (1/2 second)

8. Once the navigation query is done, the tool has to break up the GIS query into blocks of 90 HUC12s at a time. This is required because the ESRI ArcGIS Server has a limit to the allowable query size and through testing it was found that blocks of 90 HUC12 is small enough to avoid HTTP 500 errors that occur when oversized queries are sent to ESRI ArcGIS Server.

9. The blocks are sent using the query syntax HUC12 in () syntax with the 'in' array containing 90 HUCs. See this example which uses GET
   https://enviroatlas.epa.gov/arcgis/rest/services/Other/HydrologicUnits/MapServer/4/query?f=json&where=HUC_12%20in%20(%27111102050303%27%2C%27111102050301%27%2C%27111102050302%27%2C%27111102050304%27%2C%27111102050305%27%2C%27111102060506%27%2C%27111102060505%27%2C%27111102060504%27%2C%27111102060503%27%2C%27111102060502%27%2C%27111102060407%27%2C%27111102060405%27%2C%27111102060406%27%2C%27111102060402%27%2C%27111102060403%27%2C%27111102060401%27%2C%27111102060404%27%2C%27111102060501%27%2C%27111102060305%27%2C%27111102060302%27%2C%27111102060301%27%2C%27111102060205%27%2C%27111102060206%27%2C%27111102060204%27%2C%27111102060202%27%2C%27111102060203%27%2C%27111102060107%27%2C%27111102060106%27%2C%27111102060105%27%2C%27111102060103%27%2C%27111102060104%27%2C%27111102060101%27%2C%27111102060102%27%2C%27111102060201%27%2C%27111102060303%27%2C%27111102060304%27%2C%27111102070106%27%2C%27111102070102%27%2C%27111102070101%27)&returnGeometry=true&spatialRel=esriSpatialRelIntersects&outFields=*

10. The GIS queries run asynchronously

11. In the browser developer tools 'Network' tab you can see the queries being sent to the ESRI ArcGIS Server and the stall (queue) and response time.

12. at some point which varies test-by-test, the ESRI ArcGIS queries start to fail with no response

13. during additional tests, the ESRI ArcGIS Server performance gets slower and slower - probably due to some memory or CPU limitation.

Workaround:

1. If this tool is configured to use the ESRI ArcGIS Server watersgeo.epa.gov map service
   https://watersgeo.epa.gov/arcgis/rest/services/NHDPlus_NP21/WBD_NP21_Simplified/ the navigation works much quicker and does not have the errors that occur with the enviroatlas Server.

Requirements to fix problem:

1. The tool should have better handling of the error condition when the service does not return or times out. It should display an error dialog that explains what is happening.
2. There needs to be some performance tuning on the ESRI ArcGIS Server enviroatlas.epa.gov so that it at least matches the performance of the watersgeo.epa.gov server.