functional tests for process deployment

[fmigneault]

relates to #3

• add test to deploy process with CWL DockerRequirement
  • reference .cwl in href instead of ProcessDescription (see DockerRequirement note)
  • reference .cwl in owsContext within ProcessDescription -> Process
  • reference .cwl in executionUnit (combined with deploymentProfileName) (see DockerRequirement note)
• add test to deploy process with CWL WPS1Requirement (added in refactor wps-process for workflow steps #372)
  (must provide process in requirement)
  • reference .cwl in href instead of ProcessDescription
  • reference .cwl in owsContext within ProcessDescription -> Process
  • reference .cwl in executionUnit (combined with deploymentProfileName)
• add test to deploy process with WPS1 DescribeProcess endpoint
  • reference WPS1 in href instead of ProcessDescription
  • reference WPS1 in owsContext within ProcessDescription -> Process
  • reference WPS1 in executionUnit (combined with deploymentProfileName)
• add test to deploy process with WPS1 service GetCapabilities endpoint
  (must include identifier=... to derive DescribeProcess)
  • reference WPS1 in href instead of ProcessDescription
  • reference WPS1 in owsContext within ProcessDescription -> Process
  • reference WPS1 in executionUnit (combined with deploymentProfileName)
• add test to deploy process with WPS3 (REST) GET Process endpoint
  • reference WPS3 in href instead of ProcessDescription
  • reference WPS3 in owsContext within ProcessDescription -> Process
  • reference WPS3 in executionUnit (combined with deploymentProfileName)
• Deploy process directly with CWL content #434
  • package directly provided as JSON
  • reference to remote CWL package
• test with Workflow using deployed sub-processes

DockerRequirement note

Implementation missing from below tests but feature is tested across many other tests since it is the most common use case.
Should implement just to make sure it is always evaluated for consistency.

weaver/tests/wps_restapi/test_processes.py

Lines 627 to 641 in 107fec5

	# FIXME: implement
	@pytest.mark.skip(reason="not implemented")
	def test_deploy_process_CWL_DockerRequirement_href(self):
	raise NotImplementedError
	# FIXME: implement
	@pytest.mark.skip(reason="not implemented")
	def test_deploy_process_CWL_DockerRequirement_owsContext(self):
	raise NotImplementedError
	# FIXME: implement
	@pytest.mark.skip(reason="not implemented")
	def test_deploy_process_CWL_DockerRequirement_executionUnit(self):
	raise NotImplementedError

[dbyrns]

For the workflow let's start with something basic : subsetting -> climate index
For subsetting we will start using WPS1 flyingpigeon subset_bbox and when available, try to use the CWT one.

App package should be created from the following endpoints:

• https://colibri.crim.ca/twitcher/ows/proxy/flyingpigeon/wps?service=WPS&request=describeprocess&version=1.0.0&identifier=subset_bbox
• https://finch.crim.ca/wps?service=WPS&request=describeprocess&version=1.0.0&identifier=ice_days

Subsetting request sample :
https://colibri.crim.ca/twitcher/ows/proxy/flyingpigeon/wps?service=WPS&request=execute&version=1.0.0&identifier=subset_bbox&storeExecuteResponse=true&DataInputs=resource=https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/fileServer/birdhouse/nrcan/nrcan_canada_daily_v2/tasmax/nrcan_canada_daily_tasmax_2017.nc;lat0=43;lat1=50;lon0=-80;lon1=-65

(output for reference: https://colibri.crim.ca:443/wpsoutputs/flyingpigeon/81bb1fa6-4fef-11e9-9535-0242ac12000c/result_2019-03-26T17:49:45Z__4zz8XW.json)

Climate index request sample :
https://finch.crim.ca/wps?service=WPS&request=execute&version=1.0.0&identifier=ice_days&storeExecuteResponse=true&DataInputs=tasmax=files@xlink:href=https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/NRCANdaily/nrcan_canada_daily_tasmax_1990.nc;freq=MS

[dbyrns]

@davidcaron once your CWT implementation ready, you should be able to contribute to this functional tests issue and add, as discussed, a subsetter using the ESGF backend.

[fmigneault]

@dbyrns
using the above referenced fix + commit 077d9a8 + following wps_providers.yml config, WPS-2 processes are deployed successfully with their CWL WPS1Requirement on startup.

providers:
  - url: https://colibri.crim.ca/twitcher/ows/proxy/flyingpigeon/wps?service=WPS&request=describeprocess&version=1.0.0&identifier=subset_bbox
    name: colibri-flyingpieon
  - url: https://finch.crim.ca/wps?service=WPS&request=describeprocess&version=1.0.0&identifier=ice_days
    name: finch

[dbyrns]

Just be careful with the naming scheme... what we deploy in Weaver are processes not providers.
I would prefer to see a wps_processes.yml config file looking like this to avoid confusion :

processes:
  - url: https://colibri.crim.ca/twitcher/ows/proxy/flyingpigeon/wps?service=WPS&request=describeprocess&version=1.0.0&identifier=subset_bbox
    name: colibri-flyingpieon-subset_bbox
  - url: https://finch.crim.ca/wps?service=WPS&request=describeprocess&version=1.0.0&identifier=ice_days
    name: finch-ice_days