OGC API Deploy and Undeploy routes

[bbrauzzi]

Context:

The Core OGC API - Processes includes the following routes:

• GET /processes
  Lists the processes this API offers.

• GET /processes/{process-id}
  Returns a detailed description of a process.

• POST /processes/{process-id}/execution
  Executes a process (asynchronous execution will create a new job). Inputs and outputs can be specified in a JSON document that needs to be sent in the POST body.

In addition to the Core section, the OGC API - Processes series also include an extension called "OGC API - Processes - Part 2: Deploy, Replace, Undeploy (DRU)" which provides the ability to deploy, replace and undeploy processes, using an OGC Application Package definition containing the execution unit instructions for running deployed processes.
For further information please refer to https://github.com/opengeospatial/ogcapi-processes/tree/master/extensions/deploy_replace_undeploy

Proposed Changes:

In compliance with the OGC API - Processes standards, this pull-request proposes to add to the ZOO-Project the following two routes:

• POST /processes
  Deploys a process
  -- The body of the request should be an OGC Application Package definition (CWL) containing the execution unit instructions for running deployed processes.
  -- A successful execution of the operation shall be reported as a response with a HTTP status code 201.
  -- The response with HTTP status code 201 SHALL include a Location header with the URI of the newly added processes.

• DELETE /processes/{process-id}
  Undeploys a process
  -- A successful execution of the operation shall be reported as a response with a HTTP status code 204.

The deploy and undeploy routes can be configured in the main.cfg file as follows:

[servicesNamespace]  
deploy_service_provider=DeployPy  
undeploy_service_provider=UndeployPy

If the deploy_service_provider and undeploy_service_provider are not set, the routes will return
a response with HTTP status code 501 Not Implemented and body with the following message:

{
"title": "NotImplemented",
"detail": "The request method used to access the current path is not supported."
}

The pull request proposes the implementation of the routes and not of the deploy and undeploy services itself. Yet, for testing the routes two mock services have been added to the default ZOO-Project services. These mock services just perform a virtual execution but the responses are those of a successful deploy or undeploy execution.

zoo-services
├── undeploy-py
│   └── cgi-env
│       ├── UndeployPy.zcfg
│       └── undeploy_service.py
└── deploy-py
    └── cgi-env
        ├── DeployPy.zcfg
        └── deploy_service.py

How to use

Deploy

• For deploying a process an application package is needed. This CWL format document describes the inputs and outputs of a process. For more information please refer to https://www.commonwl.org/

• For this example we will use the application package of the s-expression app.
  To deploy the process run the following curl command:

   curl --location --request POST 'http://localhost/ogc-api/processes/' \
   --header 'Content-Type: application/cwl' \
   --data-raw '$graph:
   - baseCommand: s-expression
     class: CommandLineTool
     hints:
       DockerRequirement:
         dockerPull: eoepca/s-expression:dev0.0.2
     id: clt
     inputs:
       input_reference:
         inputBinding:
           position: 1
           prefix: --input_reference
         type: Directory
       s_expression:
         inputBinding:
           position: 2
           prefix: --s-expression
         type: string
       cbn:
         inputBinding:
           position: 3
           prefix: --cbn
         type: string
     outputs:
       results:
         outputBinding:
           glob: .
         type: Directory
     requirements:
       EnvVarRequirement:
         envDef:
           PATH: /srv/conda/envs/env_app_snuggs/bin:/srv/conda/envs/env_app_snuggs/bin:/srv/conda/bin:/srv/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
       ResourceRequirement: {}
     #stderr: std.err
     #stdout: std.out
   - class: Workflow
     doc: Applies s expressions to EO acquisitions
     id: s-expression
     inputs:
       input_reference:
         doc: Input product reference
         label: Input product reference
         type: Directory
       s_expression:
         doc: s expression
         label: s expression
         type: string
       cbn:
         doc: Common band name
         label: Common band name
         type: string
     label: s expressions
     outputs:
     - id: wf_outputs
       outputSource:
       - step_1/results
       type: Directory
     steps:
       step_1:
         in:
           input_reference: input_reference
           s_expression: s_expression
           cbn: cbn
         out:
         - results
         run: '\''#clt'\''
   $namespaces:
     s: https://schema.org/
   cwlVersion: v1.0
   s:softwareVersion: 0.0.2
   schemas:
   - http://schema.org/version/9.0/schemaorg-current-http.rdf'
  

  The default mock deploy service should return a response with a HTTP status code 201 with the following body

   {
   "operation": "deploy",
   "status": "success",
   "applicationPackage": "$graph:\n- baseCommand: s-expression\n class: CommandLineTool\n hints:\n DockerRequirement:\n dockerPull: eoepca/s-expression:dev0.0.2\n id: clt\n inputs:\n input_reference:\n inputBinding:\n position: 1\n prefix: --input_reference\n type: Directory\n s_expression:\n inputBinding:\n position: 2\n prefix: --s-expression\n type: string\n cbn:\n inputBinding:\n position: 3\n prefix: --cbn\n type: string\n outputs:\n results:\n outputBinding:\n glob: .\n type: Directory\n requirements:\n EnvVarRequirement:\n envDef:\n PATH: /srv/conda/envs/env_app_snuggs/bin:/srv/conda/envs/env_app_snuggs/bin:/srv/conda/bin:/srv/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n ResourceRequirement: {}\n #stderr: std.err\n #stdout: std.out\n- class: Workflow\n doc: Applies s expressions to EO acquisitions\n id: s-expression\n inputs:\n input_reference:\n doc: Input product reference\n label: Input product reference\n type: Directory\n s_expression:\n doc: s expression\n label: s expression\n type: string\n cbn:\n doc: Common band name\n label: Common band name\n type: string\n label: s expressions\n outputs:\n - id: wf_outputs\n outputSource:\n - step_1/results\n type: Directory\n steps:\n step_1:\n in:\n input_reference: input_reference\n s_expression: s_expression\n cbn: cbn\n out:\n - results\n run: '#clt'\n$namespaces:\n s: https://schema.org/\ncwlVersion: v1.0\ns:softwareVersion: 0.0.2\nschemas:\n- http://schema.org/version/9.0/schemaorg-current-http.rdf"
   }
  

and a Location header with the value http://localhost/ogc-api/processes/myNewService where myNewService is the process id defined in the Deploy service provider.

Undeploy

• To undeploy a process, run the following curl command where myNewService is the id of process you want to undeploy:

   curl --request DELETE 'http://localhost/ogc-api/processes/myNewService'
  

• The default mock undeploy service should return a response with a HTTP status code 204.

[gfenoy]

Thanks a lot for such a contribution.

I would like to say that I personally like a lot the way you implemented the new capabilities by invoking a specific service rather than integrating everything inside the ZOO-Kernel.

When I tried accessing the Basic HTML UI provided with the ZOO-Project, I noticed that the server response header were wrong, meaning that the status was not properly returned. So I applied the following patch on the response_print.c file:

@@ -3182,9 +3182,9 @@ void* printRawdataOutput(maps* conf,maps* outputs){
 
   map* status = getMapFromMaps(conf,"headers","Status");
   if(status!=NULL){
-        printf("Status: %s;\r\n",status->value);
+        printf("Status: %s\r\n\r\n",status->value);
   } else {
-        printf("Status: 200 OK;\r\n");
+        printf("Status: 200 OK\r\n\r\n");
   }
 
   // checking if the deploy service has returned the service id

This solved the issue with the server response headers (maybe the Status support should also be added to the printHeaders function).

When I was able to interact properly with the ZOO-Kernel, I decided to follow your instructions for deploying a service. This worked perfectly fine.

Nevertheless, after having deployed the service, I was unable to get the description for this service. So, I am wondering what is the purpose of deploying a service if this service cannot be accessed through traditional operations after it has been deployed. Indeed, I was supposing that I shall be able to access the Process Description and Execute the service but it seems that there is still some missing part. Probably, a ZOO-Kernel CWL internal support which would be able to parse the CWL and produce a Process Description out of it then, a way to execute such a CWL definition from the ZOO-Kernel as a traditional ZOO-Service.

I hope you can provide some clarification here.