Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Handle WPS-1.x/2.x execute request as WPS-3 'job' #126

Closed
fmigneault opened this issue May 7, 2020 · 0 comments · Fixed by #214
Closed

[Feature] Handle WPS-1.x/2.x execute request as WPS-3 'job' #126

fmigneault opened this issue May 7, 2020 · 0 comments · Fixed by #214
Assignees
Labels
process/wps1 Issue related to WPS 1.x processes support process/wps2 Issue related to WPS 2.x processes support process/wps3 Issue related to WPS 3.x (REST-JSON) processes support project/DACCS Related to DACCS project (https://github.com/orgs/DACCS-Climate) triage/feature New requested feature.

Comments

@fmigneault
Copy link
Collaborator

fmigneault commented May 7, 2020

Route /ows/wps (default path) offers the WPS-1.x/2.x XML interface (maybe also its 2.0 JSON interface with #125).

Since this route is served as a standard PyWPS service, it is possible to call an Execute request (either with GET/POST variants), but this will run the process directly on an API 'manager' thread instead of some dedicated celery worker thread/docker image. This both consumes the few resources for handling REST-API requests and possibly limit how fast/many job request the actual API can process, as each 'manager' thread becomes synchronously blocked/used until the corresponding process completes (could be a very long workflow).

Even worst, because the WPS process is executed directly from the PyWPS service, no actual job gets registered in Weaver and it is therefore impossible to obtain any feedback from the WPS 3.x routes. The only access to process status is via the base PyWPS status URL, which will consume yet another API-manager thread. Finally, when the process completes, no log/exceptions/results will be available, except the bare WPS Status route.

Therefore, WPS-1/2 Execute requests should be redirected to the corresponding /processes/{id}/jobs call, so that the job gets created and processed using the normal celery worker that will pick it up eventually. Feature #21 must be implemented at the same time as the redirect will have to work the other way around of current implementation (ie: WPS-1/2 Execute => WPS-REST 'celery' execute).
Without simultaneous implementation of the features, we will generate a circular loop of execute calls between implementations.

Given that redirections between WPS-1/2 and WPS-3 calls would be implemented here, features in #125 could be done at the same time as it implies similar requirements.

Relates to double-layer monitoring #21.

@fmigneault fmigneault added process/wps1 Issue related to WPS 1.x processes support process/wps2 Issue related to WPS 2.x processes support triage/feature New requested feature. process/wps3 Issue related to WPS 3.x (REST-JSON) processes support labels May 7, 2020
@fmigneault fmigneault self-assigned this May 7, 2020
@fmigneault fmigneault added the project/DACCS Related to DACCS project (https://github.com/orgs/DACCS-Climate) label May 11, 2020
fmigneault added a commit that referenced this issue Feb 2, 2021
@fmigneault fmigneault mentioned this issue Feb 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
process/wps1 Issue related to WPS 1.x processes support process/wps2 Issue related to WPS 2.x processes support process/wps3 Issue related to WPS 3.x (REST-JSON) processes support project/DACCS Related to DACCS project (https://github.com/orgs/DACCS-Climate) triage/feature New requested feature.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant