Increase Timeout to 2 minutes

[rajadain]

Overview

The longer the timeouts are, the more the likelihood of our resources being caught up and overloaded. To prevent that, we pick a short timeout. But, if the timeout is too short, then the expected requests will not complete.

Previously, we had set the default timeout to 1 minute in 26771b9 as part of #87. This was to allow the subbasin query for the Schuylkill HUC-8, which has 61 HUC-12s inside it, to complete. When running the geoprocessing service natively, using ./scripts/server, this would take ~13s:

61 HUC12 MultiOperation Subbasin, run 1 -> 18.611194 s
61 HUC12 MultiOperation Subbasin, run 2 -> 11.976093 s
61 HUC12 MultiOperation Subbasin, run 3 -> 14.061069 s
61 HUC12 MultiOperation Subbasin, run 4 -> 13.024482 s
61 HUC12 MultiOperation Subbasin, run 5 -> 12.124294 s

61 HUC12 MultiOperation Subbasin average -> 13.959426400000002 s

When running inside the Worker VM, with fewer resources than in native environments, it would take a little longer, ~20s:

61 HUC12 MultiOperation Subbasin, run 1 -> 21.185288 s
61 HUC12 MultiOperation Subbasin, run 2 -> 19.080413 s
61 HUC12 MultiOperation Subbasin, run 3 -> 19.835294 s
61 HUC12 MultiOperation Subbasin, run 4 -> 19.566461 s
61 HUC12 MultiOperation Subbasin, run 5 -> 19.327254 s

61 HUC12 MultiOperation Subbasin average -> 19.798942 s

With the usage of NHD Hi Res streams as of WikiWatershed/model-my-watershed#3432, the expected times have increased significantly, beyond the 1 minute timeout.

By upping the limit to 2 minutes, we ensure that the Schuylkill HUC-8's subbasin query completes successfully.

Connects WikiWatershed/model-my-watershed#3446

Demo

Schuylkill HUC-8 Subbasin with NHD Hi Res Streams, running natively:

61 HUC12 MultiOperation Subbasin, run 1 -> 68.708514 s
61 HUC12 MultiOperation Subbasin, run 2 -> 68.344673 s
61 HUC12 MultiOperation Subbasin, run 3 -> 66.588282 s
61 HUC12 MultiOperation Subbasin, run 4 -> 68.047426 s
61 HUC12 MultiOperation Subbasin, run 5 -> 67.541642 s

61 HUC12 MultiOperation Subbasin average -> 67.84610740000001 s

Schuylkill HUC-8 Subbasin with NHD Hi Res Streams, running in the Worker VM:

61 HUC12 MultiOperation Subbasin, run 1 -> 98.325671 s
61 HUC12 MultiOperation Subbasin, run 2 -> 93.014152 s
61 HUC12 MultiOperation Subbasin, run 3 -> 88.35564 s
61 HUC12 MultiOperation Subbasin, run 4 -> 91.579747 s
61 HUC12 MultiOperation Subbasin, run 5 -> 98.874168 s

61 HUC12 MultiOperation Subbasin average -> 94.0298756 s

Testing Instructions

• Ensure you have pipenv installed
• Checkout this branch and run ./scripts/setup
• Delete / comment out these lines:
  

  mmw-geoprocessing/scripts/benchmark

  Lines 61 to 68 in 49a84f2

  	echo "Timing 61 HUC12 Subbasin, separate requests per operation ->"
  	echo
  	for i in `seq 1 5`; do
  	{ time -p pipenv run python benchmark-subbasin > /dev/null; } 2>&1 |\
  	grep real |\
  	awk "{print \"61 HUC12 Subbasin, 7 operations, run $i -> \" \$2 \" s\"}"
  	done
  	echo

• Run ./scripts/server
• In another terminal window, run ./scripts/benchmark --subbasin
  • Ensure all requests complete, and are not cut off at 59 seconds

[rajadain]

In the interest of time I'm going to merge this in directly.