Merge pull request bird-house#247 from bird-house/weaver-publish-process

## Changes: - Magpie/Twitcher: update `magpie` service from [3.21.0](https://github.com/Ouranosinc/Magpie/tree/3.21.0) to [3.26.0](https://github.com/Ouranosinc/Magpie/tree/3.26.0) and bundled `twitcher` from [0.6.2](https://github.com/bird-house/twitcher/tree/v0.6.2) to [0.7.0](https://github.com/bird-house/twitcher/tree/v0.7.0). - Adds [Service Hooks](https://pavics-magpie.readthedocs.io/en/latest/configuration.html#service-hooks) allowing Twitcher to apply HTTP pre-request/post-response modifications to requested services and resources in accordance to `MagpieAdapter` implementation and using plugin Python scripts when matched against specific request parameters. - Using *Service Hooks*, inject ``X-WPS-Output-Context`` header in Weaver job submission requests through the proxied request by Twitcher and `MagpieAdapter`. This header contains the user ID that indicates to Weaver were to store job output results, allowing to save them in the corresponding user's workspace directory under `wpsoutputs` path. More details found in PR bird-house#244. - Using *Service Hooks*, filter processes returned by Weaver in JSON response from ``/processes`` endpoint using respective permissions applied onto each ``/processes/{processID}`` for the requesting user. Users will only be able to see processes for which they have read access to retrieve the process description. More details found in PR bird-house#245. - Using *Service Hooks*, automatically apply permissions for the user that successfully deployed a Weaver process using ``POST /processes`` request, granting it direct access to this process during process listing, process description request and for submitting job execution of this process. Only this user deploying the process will have access to it until further permissions are added in Magpie to share or publish it with other users, groups and/or publicly. The user must have the necessary permission to deploy a new process in the first place. More details found in PR bird-house#247.
Ouranosinc · Jun 8, 2022 · 9bfa486 · 9bfa486
2 parents 6735592 + c67c003
commit 9bfa486
Show file tree

Hide file tree

Showing 9 changed files with 377 additions and 15 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.18.13
+current_version = 1.19.0
 commit = True
 tag = False
 tag_name = {new_version}

diff --git a/.gitignore b/.gitignore
@@ -19,3 +19,7 @@ ubuntu-bionic-18.04-cloudimg-console.log
 
 ## vim temp files
 *.swp
+
+## Python temp files
+**/__pycache__
+**/*.py[cod]
diff --git a/CHANGES.md b/CHANGES.md
@@ -16,6 +16,38 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[1.19.0](https://github.com/bird-house/birdhouse-deploy/tree/1.19.0) (2022-06-08)
+------------------------------------------------------------------------------------------------------------------
+
+## Changes:
+
+- Magpie/Twitcher: update `magpie` service
+  from [3.21.0](https://github.com/Ouranosinc/Magpie/tree/3.21.0)
+  to [3.26.0](https://github.com/Ouranosinc/Magpie/tree/3.26.0) and
+  bundled `twitcher` from [0.6.2](https://github.com/bird-house/twitcher/tree/v0.6.2)
+  to [0.7.0](https://github.com/bird-house/twitcher/tree/v0.7.0).
+
+  - Adds [Service Hooks](https://pavics-magpie.readthedocs.io/en/latest/configuration.html#service-hooks) allowing 
+    Twitcher to apply HTTP pre-request/post-response modifications to requested services and resources in accordance
+    to `MagpieAdapter` implementation and using plugin Python scripts when matched against specific request parameters.
+
+  - Using *Service Hooks*, inject ``X-WPS-Output-Context`` header in Weaver job submission requests through the proxied
+    request by Twitcher and `MagpieAdapter`. This header contains the user ID that indicates to Weaver were to store 
+    job output results, allowing to save them in the corresponding user's workspace directory under `wpsoutputs` path.
+    More details found in PR https://github.com/bird-house/birdhouse-deploy/pull/244.
+
+  - Using *Service Hooks*, filter processes returned by Weaver in JSON response from ``/processes`` endpoint using
+    respective permissions applied onto each ``/processes/{processID}`` for the requesting user. Users will only be able
+    to see processes for which they have read access to retrieve the process description.
+    More details found in PR https://github.com/bird-house/birdhouse-deploy/pull/245.
+
+  - Using *Service Hooks*, automatically apply permissions for the user that successfully deployed a Weaver process 
+    using ``POST /processes`` request, granting it direct access to this process during process listing, process 
+    description request and for submitting job execution of this process.
+    Only this user deploying the process will have access to it until further permissions are added in Magpie to share
+    or publish it with other users, groups and/or publicly. The user must have the necessary permission to deploy a new
+    process in the first place. More details found in PR https://github.com/bird-house/birdhouse-deploy/pull/247.
+
 [1.18.13](https://github.com/bird-house/birdhouse-deploy/tree/1.18.13) (2022-06-07)
 ------------------------------------------------------------------------------------------------------------------
 
@@ -3351,4 +3383,3 @@ Prior Versions
 All versions prior to [1.7.0](https://github.com/bird-house/birdhouse-deploy/tree/1.7.0) were not officially tagged.
 Is it strongly recommended employing later versions to ensure better traceability of changes that could impact behavior
 and potential issues on new server instances. 
-
diff --git a/README.rst b/README.rst
@@ -14,13 +14,13 @@ for a full-fledged production platform.
     * - releases
       - | |latest-version| |commits-since|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.18.13.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.19.0.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.18.13...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.19.0...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-1.18.13-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-1.19.0-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.18.13
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.19.0
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

diff --git a/birdhouse/components/weaver/config/magpie/config.yml.template b/birdhouse/components/weaver/config/magpie/config.yml.template
@@ -10,6 +10,35 @@ providers:
     c4i: false
     type: api   # FIXME: 'ades' when https://github.com/Ouranosinc/Magpie/issues/360 implemented
     sync_type: api
+    # hook locations should be relative to mounted Twitcher location as they are run within that container
+    # see following for hooks details:
+    # - https://github.com/Ouranosinc/Magpie/blob/master/config/providers.cfg
+    # - https://pavics-magpie.readthedocs.io/en/latest/configuration.html#service-hooks
+    hooks:
+      # when a job is created in weaver, apply the header that will nest output results under user's context directory
+      # see also:
+      # - https://pavics-weaver.readthedocs.io/en/latest/processes.html?highlight=x-wps-output-context#outputs-location
+      # each path below are equivalents, but with more or less specific reference to the requested service/process
+      - type: request
+        path: "/providers/[\\w_-]+/processes/[\\w_-]+/jobs"
+        method: POST
+        target: /opt/birdhouse/src/magpie/hooks/weaver_hooks.py:add_x_wps_output_context
+      - type: request
+        path: "/processes/[\\w_-]+/jobs"
+        method: POST
+        target: /opt/birdhouse/src/magpie/hooks/weaver_hooks.py:add_x_wps_output_context
+      - type: request
+        path: "/jobs"
+        method: POST
+        target: /opt/birdhouse/src/magpie/hooks/weaver_hooks.py:add_x_wps_output_context
+      - type: response
+        path: "/processes"
+        method: GET
+        target: /opt/birdhouse/src/magpie/hooks/weaver_hooks.py:filter_allowed_processes
+      - type: response
+        path: "/processes"
+        method: POST
+        target: /opt/birdhouse/src/magpie/hooks/weaver_hooks.py:allow_user_deployed_processes
 
   # FIXME: remove when https://github.com/Ouranosinc/Magpie/issues/360 implemented, see 'default.env'
   ${WEAVER_WPS_NAME}:
@@ -66,9 +95,10 @@ permissions:
     action: create
 
   # Process deployment (write) and listing (read)
+  # use 'read-match' to allow only listing, and not describe underlying processes (require 'read' on them individually)
   - service: ${WEAVER_MANAGER_NAME}
     resource: /processes    # GET is processes listing, POST is deploy: only allow view by anonymous
-    permission: read        # under '/processes/...', JSON 'DescribeProcess', POST job submit, GET results, etc.
+    permission: read-match  # under '/processes/...', JSON 'DescribeProcess', POST job submit, GET results, etc.
     group: anonymous
     action: create