Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor wps-process for workflow steps #372

Merged
merged 43 commits into from
Jan 12, 2022
Merged

Conversation

fmigneault
Copy link
Collaborator

@fmigneault fmigneault commented Nov 25, 2021

Summary

Changes

  • No change.
  • Refactor Workflow operation flow to reuse shared input and output staging operations between implementations.
    Each new step process implementation now only requires to implement the specific operations related to deployment,
    execution, monitoring and result retrieval for their process, without need to consider Workflow intermediate staging
    operations to transfer files between steps.
  • Refactor Wps1Process and Wps3Process step processes to follow new workflow operation flow.
  • Add builtin process file_index_selector that allows the selection of a specific file within an array of files.
  • Add tests to validate chaining of Workflow steps using different combinations of process types
    including WPS-1, OGC-API and builtin implementations.
  • Move CWL script examples in documentation to separate package files in order to directly reference them in
    tests validating their deployment and execution requests.
  • Move all tests/functional/application-packages definitions into distinct directories to facilitate categorization
    of corresponding deployment, execution and package contents, and better support the various Workflow testing location
    of those files with backward compatibility.

Fixes

  • No change.
  • Fix handling of CWL Workflow outputs between steps when nested glob output binding are employed
    (resolves CWL workflow outputs within nested binding not handled #371).
  • Fix resolution of builtin process Python reference when executed locally within a Workflow step.
  • Fix resolution of process type WPS-1 from its package within a Workflow step executed as OGC-API process.
  • Fix resolution of WPS1Requirement directly provided as CWL execution unit within the deployment body.
  • Fix deployment body partially dropping invalid executionUnit sub-fields causing potential misinterpretation
    of the intended application package.
  • Fix resolution of format reference within CWL I/O record after interpretation of the loaded application package.
  • Fix missing WPS endpoint responses in generated OpenAPI for ReadTheDocs documentation.
  • Fix and improve multiple typing definitions.

@github-actions github-actions bot added ci/doc Issue related to documentation of the package feature/CWL Issue related to CWL support process/workflow Related to a Workflow process. process/wps1 Issue related to WPS 1.x processes support labels Nov 25, 2021
@github-actions github-actions bot added the ci/tests Tests of the package and features label Dec 9, 2021
@github-actions github-actions bot added feature/db Related to database or datatype manipulation. feature/job Issues related to job execution, reporting and logging. feature/oas Issues related to OpenAPI specifications. process/wps3 Issue related to WPS 3.x (REST-JSON) processes support labels Dec 9, 2021
@github-actions github-actions bot added the process/builtin Issue related to builtin application processes label Dec 16, 2021
@github-actions github-actions bot added the feature/cli Issues or features related to CLI operations. label Dec 16, 2021
CHANGES.rst Show resolved Hide resolved
Copy link
Contributor

@dbyrns dbyrns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a very welcomed refactor!

dbyrns
dbyrns previously approved these changes Jan 11, 2022
Copy link
Contributor

@dbyrns dbyrns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that you'll fix the monitor interface implementation (bool part), there is nothing more blocking this PR.

dbyrns
dbyrns previously approved these changes Jan 11, 2022
@github-actions github-actions bot added the ci/operations Related to CI operations (actions, execution, install, builds, etc.) label Jan 11, 2022
@fmigneault fmigneault merged commit 767e8b0 into master Jan 12, 2022
@fmigneault fmigneault deleted the fix-workflow-output-binding branch January 12, 2022 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci/doc Issue related to documentation of the package ci/operations Related to CI operations (actions, execution, install, builds, etc.) ci/tests Tests of the package and features feature/cli Issues or features related to CLI operations. feature/CWL Issue related to CWL support feature/db Related to database or datatype manipulation. feature/job Issues related to job execution, reporting and logging. feature/oas Issues related to OpenAPI specifications. process/builtin Issue related to builtin application processes process/workflow Related to a Workflow process. process/wps1 Issue related to WPS 1.x processes support process/wps3 Issue related to WPS 3.x (REST-JSON) processes support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CWL workflow outputs within nested binding not handled
2 participants