CLI: support execute from workflow.yaml

[josephjclark]

Overview

The OpenFn CLI currently supports running jobs through a job/expression file (ie, openfn job.js) or a workflow file (ie, openfn workflow.json)

Users of Lightning (the hosted platform at openfn.org) can sync their projects (including multiple workflows) with github. This in effect allows them to write and test their workflows locally, commit them to a github repo, and deploy that repo to Lightning. See the docs at: https://docs.openfn.org/documentation/link-to-Github

The project is saved locally in a file called project.yaml. Unfortunately, users cannot yet execute a workflow inside a project.yaml using the CLI.

Suggested Fix

It should be fairly straightforward for the CLI to import a project.yaml file, convert it into a standard Workflow object, and then execute it as usual.

Since a project.yaml file can contain multiple workflows, the CLI should support a --workflow (shorthanded to -w) argument which specifies the workflow by id. If the project only contains one workflow, we can just go ahead and use that without the -w argument.

Implementation Notes

Within the CLI and the kit repo, workflows are often known as Execution Plans. Technically an Execution Plan is a wrapper around a workflow which can include options (and maybe later metadata). Tthe terms workflow/excution plan/xplan and plan are used pretty much interchangably - this was to help divorce the platform's Workflow concept from the CLI's equivalent, which used to be a little different.

This fix should be implemented in packages/cli, integrated into the existing execute command.

There is a file in src/utils/load-plan.ts which is designed to load a workflow from a .js or .json file. You should extend the main loadPlan() to support a projectPath option (see src/options.ts::export const inputPath)

You should use the yaml npm package to handle yaml file parsing (pnpm add yaml)

See the lexicon package for type definitions for ExecutionPlan, which is the workflow structure you need to generate from the yaml.

You can see some project.yaml examples in this issue: #619

Any properties in project.yaml which are not reflected in workflow.json can safely be ignored.

Testing

Unit tests should be created to demonstrate that project.yaml. Adding good coverage to test/util/load-plan.test.ts should suffice.

A test should also be added to test/commands.test.ts which loads a yaml flie. These tests use a mock file system - see the other tests for a good reference pattern.

I would also love to see an integration test added to /integration-tests/cli/execute-workflow.ts. Create a project.yaml fixture, load it into your test, and assert on the result.

Other Notes

Other issues may well arise when trying to execute a project.yaml file. I think it's a fairly straightforward mapping but some unforeseen incompatibility could well crop up. In this event, please raise issues here on github (and maybe tag @josephjclark for help).

We welcome feedback on the docs inside this repo! Did the readme help you get started? Are you able to find the code you needed? Is there anything that would have made your work easier?