From 07a2e17c06e5a2e3eb9a9fa91e2916aa4a4391a5 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 12:12:35 +0100 Subject: [PATCH 01/18] update cli intro --- docs/build-for-developers/cli-intro.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index 944c7712318a..bf58e97c5c47 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -11,10 +11,10 @@ workflows directly from the command line. It’s simple to install, works on macOS, Windows, and Linux, and offers a range of functionality to enhance your developer experience with OpenFn. You can use the OpenFn CLI to: -- Securely run OpenFn steps and workflows +- Sync workflows between OpenFn and a local filesystem or GitHub +- Securely run OpenFn workflows - Troubleshoot and debug OpenFn steps -- Access adaptor documentation -- Deploy workflows to OpenFn +- Read and write Collections data --- @@ -25,7 +25,7 @@ Before you begin with the @openfn/cli, make sure to setup some key tooling: 1. **Code Editor:** Ensure you have a code editor installed on your machine. You can use popular editors like [VS Code](https://code.visualstudio.com/) or [Sublime](https://www.sublimetext.com/). -2. **Node.js:** Install Node.js (version 18 or later). For Linux, Windows, or +2. **Node.js:** Install Node.js (version 24 or later). For Linux, Windows, or macOS, use a version manager like [nvm](https://github.com/nvm-sh/nvm) or [asdf](https://asdf-vm.com/guide/getting-started.html). Or [install Node.js directly](https://kinsta.com/blog/how-to-install-node-js/) @@ -57,7 +57,7 @@ The word `openfn` will invoke the CLI. The word `test` will invoke the test command.
-Expand to see the expected output. +Expand to see the expected output ``` [CLI] ♦ Versions: From 34b321a7cfdc8a4c03303ce4ceaa5f9e9827a633 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 16:56:34 +0100 Subject: [PATCH 02/18] start docs --- docs/build-for-developers/cli-sync.md | 197 ++++++++++++++++++++++++++ sidebars-main.js | 3 +- 2 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 docs/build-for-developers/cli-sync.md diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md new file mode 100644 index 000000000000..5469e1cc5e46 --- /dev/null +++ b/docs/build-for-developers/cli-sync.md @@ -0,0 +1,197 @@ +--- +title: OpenFn Sync +sidebar_label: OpenFn Sync +slug: /sync +--- + +OpefnFn projects are fully portable, which means they can be moved to different +places. + +You can create a project in the app, download it to your local machine to +develop offline, re-upload it to the app, or even deploy it to a different +OpenFn server. + +We call this OpenFn Sync, and it's one of the most powerful features affording +by OpenFn Projects. + +## What is a Project? + +A Project basically set of Workflows (and maybe some configuration) which +solves, automates or integrates some business function. It usually lives in the +app, but it can just be some files on a file system. + +Inside the OpenFn app, a Project is a top level entity backed by a bunch of +database tables. + +The app Project includes a bunch of extra stuff: like configuring Channels, +tracking the run history, saving dataclips, and providing support from the AI +assistant. + +The Project can also exist on the file system, in which case it's a bunch of +files which can be read and executed by the CLI. This local representation of a +project is pretty bare-bones: you'll only find workflows and code here. + +Multiple related projects can exist on the file system at any one time. Each +will ive in a single project file. You can "check out" or "expand" one project +at a time to a local folder, which will create one file for each workflow and +one file for each step. + +Sometimes, We call this superset of all known, related, distributed projects a +Workspace. + +Not all artifacts of a Project are included in a sync. Generally we sync the +project's workflows and some of its options. But we do not sync any associated +data, credential values, usage history or AI sessions. + +## Authorization + +Before you use the CLI to fetch anything from the app, you'll need to provide +some authorization. + +The best way to do this is to set an environment variable called +`OPENFN_API_KEY`. Set it to the value of your +[Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). + +::: info Personal Access Tokens + +See +[Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) +for help setting up a token. + +::: + +If you connect to multiple OpenFn projects or apps, you can create a `.env` file +and set any environment variables you need there. The CLI will load this file +and report which keys its using. Values in your `.env` file will be preferred to +those defined in your system. + +You can also + +::: info + +This guide assumes you want to sync with our hosted SaaS app at app.openfn.org + +You can sync with a different OpenFn instance by setting the env var +`OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. + +::: + +## Downloading a Project + +To pull a project down from the app to your local machine, run: + +```bash +openfn project pull +``` + +After you've pulled a project for the first time, you don't need to specify the +UUID again (in any commands). + +The `project pull` command does three things: + +- If you do not have an `openfn.yaml` file, it'll create one +- It will _fetch_ your project from the app and save it into a single file at + `.projects/main@app.openfn.org.yaml` +- It will _checkout_ that project onto your file system, expanding workflows and + steps to their own files. + +This will create a file in your working directory called +`.projects/main@app.openfn.org.yaml`. + +## Understanding a Project Structure + +Pulling a project from the app will create a file structure that looks like +this: + +``` +├── openfn.yaml +├── .projects +│ ├── main@app.staging.openfn.org.yaml +└── workflows + ├── my-workflow + │ ├── my-workflow.yaml + │ ├── job.js +``` + +There are three key files to understand here: + +### project.yaml + +The project file saves a copy of the whole state of a project as saved in the +app. If you look inside you'll see the workflows represented as0 plain text. + +A project file is named like `@.yaml`. The alias is a local name +used to refer to a particular version of the project. + +The project file should not be edited locally as changes will be dropped on the +next fetch. + +You can fetch as many projects as you like, and each will be saved to its own +project.yaml file. + +The `.projects` folder can and should be committed to source control. + +### workflows + +Having your whole project inside a single file isn't actually a great way to +read or edit workflows. So the CLI can "checkout" or "expand" a project file +onto the file system. + +Checking out is the process of writing each workflow to a workflow.yaml file and +each step to a step.js file. This all lives in the `workflows` directory. + +Here you can edit files as much as you like, and changes will be tracked when +you push/deploy back to the app. + +You can only check out on project a time. This is actually great for working +with git, because you can checkout two projects on different branches and +compare/merge them against each other directly. + +### openfn.yaml + +This is a top-level configuration file + +We call this file system a Workspace (not a project, because technically it +holds multiple projects) + +At any time, one project can be "checked out" or "expanded". That means the the +contents of the `workflows` folder represents that specific project. You cannot +checkout two versions of the same workflow at once. + +- `openfn.yaml` contains settings and metadata about the checked out project +- `.projects` contains a yaml file for each project. This is a local copy of a + remote project. It should always reflect an app's view of a project. You + should generally not edit these files. +- `workflows` contains all the workflows for the checked out project. These are + compatible with the CLI. + +## Aliases + +Rather than identify a local project with a long-winded UUID or id string, you +can use an _alias_. + +Each local project is saved to a file like `main@app.openfn.org.yaml`, where the +`main` bit is the local alias of a project. + +You can pull a project and set the alias at the same time by running + +```bash +openfn project pull --alias dev +``` + +This will save the project to `dev@app.openfn.org.yaml` + +To change the alias, you can simply rename the file. Whatever goes before the +`@` will be treated as the alias. + +## Running workflows in projects + +## Deploying a Project + +## Sandboxes + +## GitHub + +## Cheatsheet + +An OpenFn project can be represented as a single yaml file diff --git a/sidebars-main.js b/sidebars-main.js index cc0bc31491ca..6cb20f464cbd 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -124,9 +124,10 @@ module.exports = { items: [ 'build-for-developers/cli-intro', 'build-for-developers/cli-usage', + 'build-for-developers/cli-sync', + 'build-for-developers/cli-collections', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', - 'build-for-developers/cli-collections', ], }, { From ceefdc4854c93806440d58708047fb7b7ff5fcd1 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 17:21:57 +0100 Subject: [PATCH 03/18] more docs --- docs/build-for-developers/cli-sync.md | 92 ++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 2 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index 5469e1cc5e46..dcafce4e74c2 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -65,7 +65,7 @@ and set any environment variables you need there. The CLI will load this file and report which keys its using. Values in your `.env` file will be preferred to those defined in your system. -You can also +You can also pass `--api-key` directly as a flag to most commands. ::: info @@ -184,14 +184,102 @@ This will save the project to `dev@app.openfn.org.yaml` To change the alias, you can simply rename the file. Whatever goes before the `@` will be treated as the alias. +## Checking Out + +You can checkout a project any time with: + +```bash +openfn project checkout +``` + +This will update your local workflows folder with the target project. + +If a checkout will cause changes to be lost (ie, you've changed a step.js file +but haven't deployed it), you will be warned. Add `--force` to ignore the +change, or run `openfn project clean` to wipe and reset the `workflows` folder. + +Checking out will only modify files managed by the CLI - basically workflow and +step files. If you have other files in the file system (like state files or test +files) they will be untouched. + ## Running workflows in projects +You can execute any workflow in the checked out project by name: + +```bash +openfn my-workflow +``` + +The CLI will find the workflow in your `workflows` folder and run it. You can +pass state via `-s` and set log levels as per usual. + +When running a workflow by name like this, you get two benefits: + +- **Credentials** are loaded automatically from the credential map in + `openfn.yaml`, so you don't need to pass `--credential-map` +- **Collections** use the server configured in `openfn.yaml`, so you don't need + to pass `--collections-endpoint` or anything. + ## Deploying a Project +To push your local changes back to the app, run: + +```bash +openfn project deploy +``` + +This will take your currently checked out project and push it to the app. It +will also report what has changed in the local project. + +Before uploading, the CLI fetches the latest version of the project from the app +and checks for **divergence** — meaning it checks whether any of your +locally-changed workflows have also been edited in the app since you last +pulled. If they have, the deploy will fail with an error to prevent you from +accidentally overwriting someone else's work. + +If you want to push anyway, pass `--force`: + +```bash +openfn project deploy --force +``` + +To preview what would change without actually uploading anything, use +`--dry-run`. This will log the final update payload that would be sent to the +app (as a JSON structure). + +You can deploy the checked out project as a new project on the target app by +adding the `--new` command. This is only available if you have superuser +privileges on the target instance. + +You can also deploy the checked out project to another app project by passing +its alias, id or uuid: + +``` +openfn project deploy main +``` + +If you currently have a development sandbox checked out, this would merge it +straight into the main app project. + +Note that you have to have fetched the target project locally before you can +deploy it. + ## Sandboxes ## GitHub ## Cheatsheet -An OpenFn project can be represented as a single yaml file +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------ | ----- | --------------------------------------- | +| `openfn project pull ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Switch to a different local project | +| `openfn project deploy` | Deploy checked-out project to the app | +| `openfn project deploy --dry-run` | Try a deploy but skip the upload step | +| `openfn project deploy --force` | Force the checked out project to be uploaded, ignoring any divergence warnings | +| `openfn ` | Run a workflow in the checked-out project | +| `openfn project clean` | Delete the `workflows` folder and all contents, then check out the project | From e5358103a5c87788c26bb020a1adc607eb0ab0c2 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:40:47 +0100 Subject: [PATCH 04/18] revision --- docs/build-for-developers/cli-sync.md | 130 ++++++++++++++++++-------- 1 file changed, 89 insertions(+), 41 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index dcafce4e74c2..87c62e03919d 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -1,6 +1,6 @@ --- title: OpenFn Sync -sidebar_label: OpenFn Sync +sidebar_label: Sync slug: /sync --- @@ -16,28 +16,40 @@ by OpenFn Projects. ## What is a Project? -A Project basically set of Workflows (and maybe some configuration) which -solves, automates or integrates some business function. It usually lives in the -app, but it can just be some files on a file system. +A Project is a set of Workflows which solves, automates or integrates some +business function. -Inside the OpenFn app, a Project is a top level entity backed by a bunch of -database tables. +A Project lives in the OpenFn app (either on the cloud Saas instance or on a +privately deployed instance ), but it also can exist as files on a file system. + +Attached to every Project is some metadata (like a name and a description) and +some configuration like Credentials and Collections. + +Inside the OpenFn app, a Project is a billable top level entity, where all the +workflows and configuration are saved to database tables. The app Project includes a bunch of extra stuff: like configuring Channels, -tracking the run history, saving dataclips, and providing support from the AI +tracking the run history, saving dataclips, and chat sessions with the AI assistant. -The Project can also exist on the file system, in which case it's a bunch of +The Project can also exist on a local file system, in which case it's a bunch of files which can be read and executed by the CLI. This local representation of a project is pretty bare-bones: you'll only find workflows and code here. Multiple related projects can exist on the file system at any one time. Each -will ive in a single project file. You can "check out" or "expand" one project +will live in a single project file. You can "check out" or "expand" one project at a time to a local folder, which will create one file for each workflow and one file for each step. -Sometimes, We call this superset of all known, related, distributed projects a -Workspace. +It's typical that a single conceptual project - that is, the code and +configuration which drives a business function - exists in several places at +once. It might have several representations in the app through sandboxes; be +backed up to GitHub; run locally on a developer's machine; and be distributed to +several remote instances to run in production. + +Sometimes, we call this superset of all known, related, distributed Projects a +Workspace. The Sync problem is the question of how code and configuration is +copied/deployed/replicated across Project instances. Not all artifacts of a Project are included in a sync. Generally we sync the project's workflows and some of its options. But we do not sync any associated @@ -52,7 +64,7 @@ The best way to do this is to set an environment variable called `OPENFN_API_KEY`. Set it to the value of your [Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). -::: info Personal Access Tokens +:::info Personal Access Tokens See [Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) @@ -67,9 +79,10 @@ those defined in your system. You can also pass `--api-key` directly as a flag to most commands. -::: info +:::info -This guide assumes you want to sync with our hosted SaaS app at app.openfn.org +This guide assumes you want to sync with our hosted SaaS app at +[app.openfn.org](https://app.openfn.org) You can sync with a different OpenFn instance by setting the env var `OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. @@ -84,19 +97,34 @@ To pull a project down from the app to your local machine, run: openfn project pull ``` +This will create a file in your working directory called +`.projects/main@app.openfn.org.yaml`. + +:::info + +Every Project in the app has a unique identifier, called a UUID, which is used +to reference it. It is a 32 digit number of the form +`a6cc5bdd-b04f-4413-b4b8-132a5115acac` + +You can copy a Project's UUID from the URL by browsing to it in the app. It's +the long string after `projects`. + +For example, the UUID is the bold section in: +{'https://openfn.org/projects/'}{'abc087dd-3963-4260-8d09-ced2e1ff2bb0'}{'/w'} + +::: + After you've pulled a project for the first time, you don't need to specify the -UUID again (in any commands). +UUID again. You can use the alias, the id, or leave the identifier blank to use +the currently checked out project. The `project pull` command does three things: - If you do not have an `openfn.yaml` file, it'll create one -- It will _fetch_ your project from the app and save it into a single file at - `.projects/main@app.openfn.org.yaml` -- It will _checkout_ that project onto your file system, expanding workflows and - steps to their own files. - -This will create a file in your working directory called -`.projects/main@app.openfn.org.yaml`. +- It will _fetch_ (download) your project from the app and save it into a single + file at `.projects/main@app.openfn.org.yaml` +- It will _checkout_ (expand) that project onto your file system, expanding + workflows and steps to their own files. ## Understanding a Project Structure @@ -118,13 +146,14 @@ There are three key files to understand here: ### project.yaml The project file saves a copy of the whole state of a project as saved in the -app. If you look inside you'll see the workflows represented as0 plain text. +app. If you look inside you'll see the workflows represented as plain text. A project file is named like `@.yaml`. The alias is a local name -used to refer to a particular version of the project. +used to refer to a particular version of the project. The domain comes from the +OpenFn instance the project was downloaded from. -The project file should not be edited locally as changes will be dropped on the -next fetch. +The project file should not be edited locally as any changes will be dropped on +the next fetch. You can fetch as many projects as you like, and each will be saved to its own project.yaml file. @@ -149,21 +178,10 @@ compare/merge them against each other directly. ### openfn.yaml -This is a top-level configuration file - -We call this file system a Workspace (not a project, because technically it -holds multiple projects) - -At any time, one project can be "checked out" or "expanded". That means the the -contents of the `workflows` folder represents that specific project. You cannot -checkout two versions of the same workflow at once. - -- `openfn.yaml` contains settings and metadata about the checked out project -- `.projects` contains a yaml file for each project. This is a local copy of a - remote project. It should always reflect an app's view of a project. You - should generally not edit these files. -- `workflows` contains all the workflows for the checked out project. These are - compatible with the CLI. +This is a top-level configuration file which can mostly be ignored. It is used +by OpenFn tooling to recognise a project root folder. It also holds +configuration options for all local projects, and metadata about the currently +checked out project. ## Aliases @@ -266,8 +284,38 @@ deploy it. ## Sandboxes +The CLI is fully compatible with sandboxes. Treat them like any other project: +fetch them for the first time with their UUID. + +Use the `checkout` command to switch between sandboxes and projects locally. +Remember you can only have a single project checked out at once. The CLI will +warn you if a checkout will cause you to lose any local changes. + +When fetching a sandbox, by default the project alias will be set to the sandbox +name. + +You can merge two projects locally with `openfn merge`, and deploy the resulting +project to the app (you'll likely have to force push the change). This is useful +for conflict resolution. + ## GitHub +You can configure a Project to sync automatically to GitHub. This means that +commit to GitHub will automatically deploy changes to an OpenFn Project; and +pressing Save & Sync in the app will commit back to GitHub. + +Under the hood, GitHub Sync uses the CLI's `pull` and `deploy` commands, +triggered by GitHub Actions, to sync your projects. + +Note that by default, GitHub Sync will use the legacy format, with `state.json`, +`project.yaml` and `config.json` files. When setting up a new GitHub Sync, you +can choose to select the v2 format. V2 sync is only suitable for pulling a +single Project per branch to github, as multiple projects will overwrite the +same `workflows/` folder. + +See [Version Control](/documentation/link-to-GitHub) for more details about +GitHub Sync. + ## Cheatsheet | Command | Description | From c68ee6921bae5a55e575f316fe58841d1d743aa8 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:43:16 +0100 Subject: [PATCH 05/18] claude typos --- docs/build-for-developers/cli-sync.md | 28 +++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index 87c62e03919d..dd147132df67 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -4,14 +4,14 @@ sidebar_label: Sync slug: /sync --- -OpefnFn projects are fully portable, which means they can be moved to different +OpenFn projects are fully portable, which means they can be moved to different places. You can create a project in the app, download it to your local machine to develop offline, re-upload it to the app, or even deploy it to a different OpenFn server. -We call this OpenFn Sync, and it's one of the most powerful features affording +We call this OpenFn Sync, and it's one of the most powerful features afforded by OpenFn Projects. ## What is a Project? @@ -19,8 +19,8 @@ by OpenFn Projects. A Project is a set of Workflows which solves, automates or integrates some business function. -A Project lives in the OpenFn app (either on the cloud Saas instance or on a -privately deployed instance ), but it also can exist as files on a file system. +A Project lives in the OpenFn app (either on the cloud SaaS instance or on a +privately deployed instance), but it also can exist as files on a file system. Attached to every Project is some metadata (like a name and a description) and some configuration like Credentials and Collections. @@ -172,7 +172,7 @@ each step to a step.js file. This all lives in the `workflows` directory. Here you can edit files as much as you like, and changes will be tracked when you push/deploy back to the app. -You can only check out on project a time. This is actually great for working +You can only check out one project at a time. This is actually great for working with git, because you can checkout two projects on different branches and compare/merge them against each other directly. @@ -197,7 +197,7 @@ You can pull a project and set the alias at the same time by running openfn project pull --alias dev ``` -This will save the project to `dev@app.openfn.org.yaml` +This will save the project to `dev@app.openfn.org.yaml`. To change the alias, you can simply rename the file. Whatever goes before the `@` will be treated as the alias. @@ -266,7 +266,7 @@ To preview what would change without actually uploading anything, use app (as a JSON structure). You can deploy the checked out project as a new project on the target app by -adding the `--new` command. This is only available if you have superuser +adding the `--new` flag. This is only available if you have superuser privileges on the target instance. You can also deploy the checked out project to another app project by passing @@ -294,7 +294,7 @@ warn you if a checkout will cause you to lose any local changes. When fetching a sandbox, by default the project alias will be set to the sandbox name. -You can merge two projects locally with `openfn merge`, and deploy the resulting +You can merge two projects locally with `openfn project merge`, and deploy the resulting project to the app (you'll likely have to force push the change). This is useful for conflict resolution. @@ -318,12 +318,12 @@ GitHub Sync. ## Cheatsheet -| Command | Description | -| ---------------------------------------- | ------------------------------------------------------------------------------ | ----- | --------------------------------------- | -| `openfn project pull ` | Pull a project from the app for the first time | -| `openfn project pull` | Re-pull the current project | -| `openfn project pull --alias dev` | Pull and set a local alias | -| `openfn project fetch ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Fetch a project without checking it out | | `openfn project` | List all local projects in the current working folder | | `openfn project checkout ` | Switch to a different local project | | `openfn project deploy` | Deploy checked-out project to the app | From ce39e8c90de44993e997fbb27cbd7f9c8d7debe6 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:59:37 +0100 Subject: [PATCH 06/18] thinking about structure --- docs/build-for-developers/cli-sync.md | 28 +- docs/deploy/portability-v1.md | 439 ++++++++++++++++++++++++++ docs/deploy/portability.md | 61 +++- 3 files changed, 498 insertions(+), 30 deletions(-) create mode 100644 docs/deploy/portability-v1.md diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index dd147132df67..fbd5399b1ffb 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -11,8 +11,8 @@ You can create a project in the app, download it to your local machine to develop offline, re-upload it to the app, or even deploy it to a different OpenFn server. -We call this OpenFn Sync, and it's one of the most powerful features afforded -by OpenFn Projects. +We call this OpenFn Sync, and it's one of the most powerful features afforded by +OpenFn Projects. ## What is a Project? @@ -126,7 +126,7 @@ The `project pull` command does three things: - It will _checkout_ (expand) that project onto your file system, expanding workflows and steps to their own files. -## Understanding a Project Structure +## Understanding Project Structure Pulling a project from the app will create a file structure that looks like this: @@ -266,8 +266,8 @@ To preview what would change without actually uploading anything, use app (as a JSON structure). You can deploy the checked out project as a new project on the target app by -adding the `--new` flag. This is only available if you have superuser -privileges on the target instance. +adding the `--new` flag. This is only available if you have superuser privileges +on the target instance. You can also deploy the checked out project to another app project by passing its alias, id or uuid: @@ -294,9 +294,9 @@ warn you if a checkout will cause you to lose any local changes. When fetching a sandbox, by default the project alias will be set to the sandbox name. -You can merge two projects locally with `openfn project merge`, and deploy the resulting -project to the app (you'll likely have to force push the change). This is useful -for conflict resolution. +You can merge two projects locally with `openfn project merge`, and deploy the +resulting project to the app (you'll likely have to force push the change). This +is useful for conflict resolution. ## GitHub @@ -318,12 +318,12 @@ GitHub Sync. ## Cheatsheet -| Command | Description | -| -------------------------------------------- | ------------------------------------------------------------------------------ | -| `openfn project pull ` | Pull a project from the app for the first time | -| `openfn project pull` | Re-pull the current project | -| `openfn project pull --alias dev` | Pull and set a local alias | -| `openfn project fetch ` | Fetch a project without checking it out | +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------ | +| `openfn project pull ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Fetch a project without checking it out | | `openfn project` | List all local projects in the current working folder | | `openfn project checkout ` | Switch to a different local project | | `openfn project deploy` | Deploy checked-out project to the app | diff --git a/docs/deploy/portability-v1.md b/docs/deploy/portability-v1.md new file mode 100644 index 000000000000..679872d94e2c --- /dev/null +++ b/docs/deploy/portability-v1.md @@ -0,0 +1,439 @@ +--- +title: Portability v1 (legacy) +--- + +The portability specification allows for the representations of entire workflow +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. + +If you're interested in contributing to the specification, reach out to OpenFn +via the [community forum](https://community.openfn.org), write to us, or suggest +changes by submitting a pull request here. + +:::warning + +This is the legacy version of the OpenFn Portability spec. + +For the latest version, see [Portability](portability) + +::: + +## Projects "as code" + +Entire projects (groups of workflows with their associated triggers, edges, +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by (a) allowing workflows to be +built and tested locally; (b) enabling project version control and an audit +trail of project changes; and (c) allowing users to port existing projects +between different instances (i.e., deployments) of Lightning. + +### Directory structure + +Many users keep OpenFn projects in git repositories, and this is a common +structure: + +``` +myProject/ +├── workflow-a/ +│ ├── job-1.js +│ ├── job-2.js +│ └── job-3.js +├── workflow-b/ +│ └── job-4.js +├── project.yaml +├── projectState.json +└── config.json +``` + +:::info Directory Structure + +There are commonly used 3 directory structure for OpenFn projects namely: +standard, production & test, and monorepo. To learn more, please see the OpenFn +[GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). + +::: + +### The project **_spec_** + +The project specification (or "spec") is often saved as a `project.yaml` file. +While most of the spec is written inline, many developers prefer to track their +job bodies in separate `.js` files and they then reference them with a relative +path. + +```yaml +name: openhie-project +description: Some sample +credentials: + jane-smith@test.com-HAPI-FHIR: + owner: jane-smith@test.com + name: HAPI FHIR +workflows: + OpenHIE-Workflow: + name: OpenHIE Workflow + jobs: + FHIR-standard-Data-with-change: + name: FHIR-standard-Data-with-change + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/my-fancy-script.js + + Send-to-OpenHIM-to-route-to-SHR: + name: Send-to-OpenHIM-to-route-to-SHR + adaptor: '@openfn/language-http@latest' + enabled: true + credential: jane-smith@test.com-HAPI-FHIR + body: | + fn(state => { + console.log("hello github integration") + return state + }); + + Notify-CHW-upload-successful: + name: Notify-CHW-upload-successful + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: fn(state => state); + + Notify-CHW-upload-failed: + name: Notify-CHW-upload-failed + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/notify-failure.js + + triggers: + webhook: + type: webhook + edges: + webhook->FHIR-standard-Data-with-change: + source_trigger: webhook + target_job: FHIR-standard-Data-with-change + condition: always + FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: + source_job: FHIR-standard-Data-with-change + target_job: Send-to-OpenHIM-to-route-to-SHR + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-successful + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-failed + condition: on_job_failure +``` + +In this spec, you can see the different ways of defining a job's body: + +1. Inline body: Used in the `FHIR-standard-Data-with-change` and + `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the + YAML file. + +2. External file reference: Used in both `Notify-CHW-upload-successful` and + `Notify-CHW-upload-failed` jobs. The body is stored in separate files, + referenced by the path key. This allows for better organization of complex + job logic. + +When using file paths: + +- Paths are relative to the location of the `project.yaml` file. +- Ensure that the referenced files exist and contain valid job body code. +- This method is particularly useful for complex jobs or when you want to reuse + job bodies across different projects. + +### The project **_state_** + +The project state is a representation of a particular project as _on a specific +Lightning instance_. It is often saved as `projectState.json` and contains UUIDs +for resources on a particular Lightning deployment. + +```json +{ + "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", + "name": "openhie-project", + "description": null, + "inserted_at": "2023-08-25T08:57:31", + "updated_at": "2023-08-25T08:57:31", + "scheduled_deletion": null, + "requires_mfa": false, + "project_credentials": { + "jane-smith@test.com-HAPI-FHIR": { + "id": "25f48989-d349-4eb8-99c3-923ebba5b116", + "name": "HAPI FHIR", + "owner": "jane-smith@test.com" + } + }, + "workflows": { + "OpenHIE-Workflow": { + "id": "27ae2937-0959-48b8-a597-b1646aae8c14", + "name": "OpenHIE Workflow", + "jobs": { + "Transform-data-to-FHIR-standard": { + "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR": { + "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "name": "Send-to-OpenHIM-to-route-to-SHR", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-successful": { + "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", + "name": "Notify-CHW-upload-successful", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-failed": { + "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", + "name": "Notify-CHW-upload-failed", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "FHIR-standard-Data": { + "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", + "delete": true + }, + "FHIR-standard-Data-with-change": { + "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "name": "FHIR-standard-Data-with-change", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + } + }, + "triggers": { + "webhook": { + "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "type": "webhook" + } + }, + "edges": { + "webhook->Transform-data-to-FHIR-standard": { + "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", + "delete": true + }, + "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { + "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { + "id": "26c12f7f-7806-4008-87cd-6747998f95f4", + "condition": "on_job_success", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { + "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", + "condition": "on_job_failure", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" + }, + "webhook->FHIR-standard-Data": { + "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", + "delete": true + }, + "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { + "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", + "delete": true + }, + "webhook->FHIR-standard-Data-with-change": { + "id": "75e7f7d8-274b-410d-9600-730bbd535229", + "condition": "always", + "source_job_id": null, + "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" + }, + "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { + "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", + "condition": "on_job_success", + "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "source_trigger_id": null, + "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" + } + } + } + } +} +``` + +## Using the CLI interact with projects + +The project spec and project state can be used for a variety of reasons, e.g. +one could generate the state and spec as backups of the project or one could +generate these files and use them for auditing and record keeping, etc. The +OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with +commands that can be used to pull project configurations down from a running +Lightning server, and to deploy or push updates to existing projects on a +Lightning server. To learn more about automated version control via pull and +deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) +docs. + +:::info Don't have the CLI yet? + +Install it by running `npm install -g @openfn/cli` + +::: + +Before using the CLI, configure it either by passing in environment variables: + +``` +OPENFN_ENDPOINT=https://app.openfn.org +OPENFN_API_KEY=yourSecretApiToken +``` + +Or through a `config.json` file: + +```json +{ + // Required, can be overridden or set with `OPENFN_API_KEY` env var + "apiKey": "***", + + // Optional: can be set using the -p, defaults to project.yaml + "specPath": "project.yaml", + + // Optional: can be set using -s, defaults to .state.json + "statePath": ".state.json", + + // Optional: defaults to OpenFn.org's API, can be overridden or set with + // `OPENFN_ENDPOINT` env var + "endpoint": "https://app.openfn.org" +} +``` + +More details on the CLI can be found +[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). + +### `openfn pull` to generate spec & state + +To generate the spec and state files for an existing project, use: + +```sh +openfn pull {YOUR-PROJECT-UUID} -c ./config.json +``` + +This command will save (or overwrite) a project spec and state file based on the +path you've set in your configuration. + +### `openfn deploy` to create new projects + +To deploy a new project to a Lightning instance from a project spec (without a +project state) file use: + +```sh +openfn deploy -c config.json +``` + +### `openfn deploy` to update existing projects + +With a valid project state defined in your `config.json`, the same +`openfn deploy` command will beam up your changes as described by a difference +between your project spec and what's found on the server. + +```sh +openfn deploy -c config.json +Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. +Project found. +[CLI] ♦ Changes: + { + workflows: [ + { + jobs: [ + { +- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" ++ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" + } + ... + ... + ... + ] + } + ] + } + +? Deploy? yes +[CLI] ♦ Deployed. +``` + +## Getting Help with the cli + +The cli package comes with an inbuilt `help`. Adding `--help` to a command such +as `openfn deploy --help` will result in a help message describing the command +and the options available when using this command. See an example below + +```sh +openfn deploy --help +openfn deploy + +Deploy a project's config to a remote Lightning instance + +Options: + --version Show version number [boolean] + --help Show help [boolean] + -c, --config, --config-path The location of your config file [default: "./.config.json"] + --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] + --describe Downloads the project yaml from the specified instance [boolean] + -l, --log Set the log level [string] + --log-json Output all logs as JSON objects [boolean] + -p, --project-path The location of your project.yaml file [string] + -s, --state-path Path to the state file +``` + +## Troubleshooting + +This section covers solutions to some errors you might come across when using +OpenFn pull or deploy in your projects. + +### Extraneous Workflow ID + +#### Description + +This error occurs when you run `openfn deploy` and there is a mismatch between +between IDs of workflows in your projectSpec and your OpenFn instance. When this +occurs, the error will be written out in an error object as shown below: + +``` +[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: +{ + "errors": { + "workflows": { + "1-ingest-messages": { + "base": [ + "extraneous parameters: workflow_id" + ] + }, + "2-calculate-indicators": { + "base": [ + "extraneous parameters: workflow_id" + ] + } + } + } +``` + +#### Solution + +Run `openfn pull` to update your local instance and keep IDs in sync, +incorporate your changes and run `openfn deploy` again. + +## Other Versions + +- [Portability Spec v2](portability-versions#v2) +- [Portability Spec v1](portability-versions#v1) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 10f9d2ac8973..1c48ecfb54d0 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -2,30 +2,58 @@ title: Portability --- -## Intent - The portability specification allows for the representations of entire workflow -projects "as code", lets user move between various deployment pathways (cloud, -local, DIY, etc.) and proposes a globally-applicable way of **_specifying -workflow automation_** and **_systems integration_** that might be applied -across workflow-engines/integration platforms across the sector. Nothing about -the spec _must_ be specific to OpenFn or any one of our individual products. We -envision a future in which software built with Lightning, the OpenFn Integration -Toolkit, and entirely new and different integration/workflow tools can adopt -this specification. +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. If you're interested in contributing to the specification, reach out to OpenFn via the [community forum](https://community.openfn.org), write to us, or suggest changes by submitting a pull request here. +:::info + +This document describes v2 of the OpenFn deployment spec, in-line with +[OpenFn Sync](/documentation/sync). + +For the v1 spec (using state, project and config files), see +[Legacy Portability](./portability-v1). + +::: + ## Projects "as code" Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. This improves the OpenFn -developer experience by (a) allowing workflows to be built and tested locally; -(b) enabling project version control and an audit trail of project changes; and -(c) allowing users to port existing projects between different instances (i.e., -deployments) of Lightning. +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by: + +1. Allowing workflows to be built and tested locally +2. Enabling project version control and an audit trail of project changes +3. Allowing users to port existing projects between different instances (i.e., + deployments) of Lightning. + +# Joe's Thoughts + +What should go in this file? + +The Sync document is the best equivalent. Should I just reference that? + +This document talks about a portability "spec". But I don't really see that +explained anywhere. Should I rewrite that stuff into a higher level manifesto? + +There's also already a v1 and v2 spec. So actually what I'm doing now is a v3? + +I should chat with Taylor about what to do here. + +I am super unconfortable having multiple versions of the sync workflow on the +site ### Directory structure @@ -395,7 +423,7 @@ OpenFn pull or deploy in your projects. ### Extraneous Workflow ID -#### Description +#### Description This error occurs when you run `openfn deploy` and there is a mismatch between between IDs of workflows in your projectSpec and your OpenFn instance. When this @@ -419,6 +447,7 @@ occurs, the error will be written out in an error object as shown below: } } ``` + #### Solution Run `openfn pull` to update your local instance and keep IDs in sync, From 003cb63343e0f475f3336bbbdd54a2eca5f3c6f0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 18:11:22 +0100 Subject: [PATCH 07/18] fiddling with sync --- docs/manage-projects/link-to-gh.md | 38 +++++++++++++++--------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/manage-projects/link-to-gh.md b/docs/manage-projects/link-to-gh.md index 92a8204f03ec..f919be00199b 100644 --- a/docs/manage-projects/link-to-gh.md +++ b/docs/manage-projects/link-to-gh.md @@ -1,31 +1,31 @@ --- -title: Version Control -sidebar_label: Version Control (GitHub Sync) +title: GitHub Sync +sidebar_label: GitHub Sync slug: /link-to-GitHub --- -The version control (GitHub Sync) feature lets users track and manage changes to -their OpenFn projects in GitHub. GitHub Sync enables a 2-way sync between your -OpenFn project and your GitHub repository. By 2-way sync, we mean that you can -**sync changes made to your project on OpenFn to GitHub**, and you can **deploy -changes you've made to your project on GitHub to OpenFn**. +GitHub Sync enables two-way sync between OpenFn project and your GitHub +repository. + +This means changes made to a Project in OpenFn can commit back to your GitHub +repo; and commits pushed to GitHub can update the Project in your OpenFn App. :::info For Cloud Hosted OpenFn Users GitHub Sync is only available in projects that are subscribed to Core, Growth, -Scale or Custom plans on Cloud Hosted OpenFn. Users can however connect their -OpenFn account to GitHub by authenticating OpenFn to access their GitHub account -by navigating to `User Profile` page and clicking ` Connect your GitHub -Account". +Scale or Custom plans on Cloud Hosted OpenFn. + +Users can however connect their OpenFn account to GitHub by authenticating +OpenFn to access their GitHub account by navigating to `User Profile` page and +clicking ` Connect your GitHub Account". ::: -### Configuring your project to use GitHub Sync +### Configuring your Project to use GitHub Sync -Users are able to configure their projects to have access to one or more -repositories on GitHub. To enable sync, the OpenFn project requires a repository -where a GitHub OpenFn application is installed and users are required to have -administrative access to the repositiory. +Users can configure their projects to access one or more repositories on GitHub. +Users must have admin access to the GitHub Repo to ensure the OpenFn application +is installed. To configure your project to use Github sync, follow the following steps: @@ -285,8 +285,6 @@ your-git-monorepo :::tip A sync in time, saves nine -#### Syncing Changes from OpenFn to GitHub - When you sync changes from OpenFn to GitHub, the `projectSpec.yaml` file in your repository will be updated with the changes made to the project in OpenFn. For a project with a directory structure that uses relative paths for job code, OpenFn @@ -298,7 +296,9 @@ When you keep job code in relative file paths, ensure to update the `projectSpec.yaml` file based on changes to the files or paths in your project repository. A GitHub action is automatically triggered to push changes to OpenFn ensuring that future syncs are not affected. Changes can include adding, -renaming, deleting a file or updating a file path. ::: +renaming, deleting a file or updating a file path. + +::: ## Troubleshooting From d57d90f6e89ba949ae89de7e1be28ce14ba934b0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 12:13:33 +0100 Subject: [PATCH 08/18] remove notes --- docs/deploy/portability.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 1c48ecfb54d0..4a89c08988b3 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -39,22 +39,6 @@ This improves the OpenFn developer experience by: 3. Allowing users to port existing projects between different instances (i.e., deployments) of Lightning. -# Joe's Thoughts - -What should go in this file? - -The Sync document is the best equivalent. Should I just reference that? - -This document talks about a portability "spec". But I don't really see that -explained anywhere. Should I rewrite that stuff into a higher level manifesto? - -There's also already a v1 and v2 spec. So actually what I'm doing now is a v3? - -I should chat with Taylor about what to do here. - -I am super unconfortable having multiple versions of the sync workflow on the -site - ### Directory structure Many users keep OpenFn projects in git repositories, and this is a common From 2be5f7f0e91a74d51cc785072d18b38f1f4c0eb0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:24:54 +0100 Subject: [PATCH 09/18] typo --- docs/deploy/portability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 4a89c08988b3..54aea57f5eac 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -3,7 +3,7 @@ title: Portability --- The portability specification allows for the representations of entire workflow -projects "as code"; lets user moves between various deployment pathways (such as +projects "as code"; lets users move between various deployment pathways (such as cloud, local, hosted); and proposes a globally-applicable way of defining workflow automation and system integration rules that might be applied across workflow-engines/integration platforms across the sector. From 76822c6447c60c72a20799393e7c5fff51cf38b2 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:26:19 +0100 Subject: [PATCH 10/18] rollback changes --- docs/build-for-developers/cli-intro.md | 10 +++---- docs/manage-projects/link-to-gh.md | 38 +++++++++++++------------- 2 files changed, 24 insertions(+), 24 deletions(-) diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index bf58e97c5c47..944c7712318a 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -11,10 +11,10 @@ workflows directly from the command line. It’s simple to install, works on macOS, Windows, and Linux, and offers a range of functionality to enhance your developer experience with OpenFn. You can use the OpenFn CLI to: -- Sync workflows between OpenFn and a local filesystem or GitHub -- Securely run OpenFn workflows +- Securely run OpenFn steps and workflows - Troubleshoot and debug OpenFn steps -- Read and write Collections data +- Access adaptor documentation +- Deploy workflows to OpenFn --- @@ -25,7 +25,7 @@ Before you begin with the @openfn/cli, make sure to setup some key tooling: 1. **Code Editor:** Ensure you have a code editor installed on your machine. You can use popular editors like [VS Code](https://code.visualstudio.com/) or [Sublime](https://www.sublimetext.com/). -2. **Node.js:** Install Node.js (version 24 or later). For Linux, Windows, or +2. **Node.js:** Install Node.js (version 18 or later). For Linux, Windows, or macOS, use a version manager like [nvm](https://github.com/nvm-sh/nvm) or [asdf](https://asdf-vm.com/guide/getting-started.html). Or [install Node.js directly](https://kinsta.com/blog/how-to-install-node-js/) @@ -57,7 +57,7 @@ The word `openfn` will invoke the CLI. The word `test` will invoke the test command.
-Expand to see the expected output +Expand to see the expected output. ``` [CLI] ♦ Versions: diff --git a/docs/manage-projects/link-to-gh.md b/docs/manage-projects/link-to-gh.md index f919be00199b..92a8204f03ec 100644 --- a/docs/manage-projects/link-to-gh.md +++ b/docs/manage-projects/link-to-gh.md @@ -1,31 +1,31 @@ --- -title: GitHub Sync -sidebar_label: GitHub Sync +title: Version Control +sidebar_label: Version Control (GitHub Sync) slug: /link-to-GitHub --- -GitHub Sync enables two-way sync between OpenFn project and your GitHub -repository. - -This means changes made to a Project in OpenFn can commit back to your GitHub -repo; and commits pushed to GitHub can update the Project in your OpenFn App. +The version control (GitHub Sync) feature lets users track and manage changes to +their OpenFn projects in GitHub. GitHub Sync enables a 2-way sync between your +OpenFn project and your GitHub repository. By 2-way sync, we mean that you can +**sync changes made to your project on OpenFn to GitHub**, and you can **deploy +changes you've made to your project on GitHub to OpenFn**. :::info For Cloud Hosted OpenFn Users GitHub Sync is only available in projects that are subscribed to Core, Growth, -Scale or Custom plans on Cloud Hosted OpenFn. - -Users can however connect their OpenFn account to GitHub by authenticating -OpenFn to access their GitHub account by navigating to `User Profile` page and -clicking ` Connect your GitHub Account". +Scale or Custom plans on Cloud Hosted OpenFn. Users can however connect their +OpenFn account to GitHub by authenticating OpenFn to access their GitHub account +by navigating to `User Profile` page and clicking ` Connect your GitHub +Account". ::: -### Configuring your Project to use GitHub Sync +### Configuring your project to use GitHub Sync -Users can configure their projects to access one or more repositories on GitHub. -Users must have admin access to the GitHub Repo to ensure the OpenFn application -is installed. +Users are able to configure their projects to have access to one or more +repositories on GitHub. To enable sync, the OpenFn project requires a repository +where a GitHub OpenFn application is installed and users are required to have +administrative access to the repositiory. To configure your project to use Github sync, follow the following steps: @@ -285,6 +285,8 @@ your-git-monorepo :::tip A sync in time, saves nine +#### Syncing Changes from OpenFn to GitHub + When you sync changes from OpenFn to GitHub, the `projectSpec.yaml` file in your repository will be updated with the changes made to the project in OpenFn. For a project with a directory structure that uses relative paths for job code, OpenFn @@ -296,9 +298,7 @@ When you keep job code in relative file paths, ensure to update the `projectSpec.yaml` file based on changes to the files or paths in your project repository. A GitHub action is automatically triggered to push changes to OpenFn ensuring that future syncs are not affected. Changes can include adding, -renaming, deleting a file or updating a file path. - -::: +renaming, deleting a file or updating a file path. ::: ## Troubleshooting From d0b39a8176fedccd55a5664ac3bfa2de87eee6ea Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:32:47 +0100 Subject: [PATCH 11/18] rollback again --- docs/build-for-developers/cli-sync.md | 333 -------------------------- sidebars-main.js | 1 - 2 files changed, 334 deletions(-) delete mode 100644 docs/build-for-developers/cli-sync.md diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md deleted file mode 100644 index fbd5399b1ffb..000000000000 --- a/docs/build-for-developers/cli-sync.md +++ /dev/null @@ -1,333 +0,0 @@ ---- -title: OpenFn Sync -sidebar_label: Sync -slug: /sync ---- - -OpenFn projects are fully portable, which means they can be moved to different -places. - -You can create a project in the app, download it to your local machine to -develop offline, re-upload it to the app, or even deploy it to a different -OpenFn server. - -We call this OpenFn Sync, and it's one of the most powerful features afforded by -OpenFn Projects. - -## What is a Project? - -A Project is a set of Workflows which solves, automates or integrates some -business function. - -A Project lives in the OpenFn app (either on the cloud SaaS instance or on a -privately deployed instance), but it also can exist as files on a file system. - -Attached to every Project is some metadata (like a name and a description) and -some configuration like Credentials and Collections. - -Inside the OpenFn app, a Project is a billable top level entity, where all the -workflows and configuration are saved to database tables. - -The app Project includes a bunch of extra stuff: like configuring Channels, -tracking the run history, saving dataclips, and chat sessions with the AI -assistant. - -The Project can also exist on a local file system, in which case it's a bunch of -files which can be read and executed by the CLI. This local representation of a -project is pretty bare-bones: you'll only find workflows and code here. - -Multiple related projects can exist on the file system at any one time. Each -will live in a single project file. You can "check out" or "expand" one project -at a time to a local folder, which will create one file for each workflow and -one file for each step. - -It's typical that a single conceptual project - that is, the code and -configuration which drives a business function - exists in several places at -once. It might have several representations in the app through sandboxes; be -backed up to GitHub; run locally on a developer's machine; and be distributed to -several remote instances to run in production. - -Sometimes, we call this superset of all known, related, distributed Projects a -Workspace. The Sync problem is the question of how code and configuration is -copied/deployed/replicated across Project instances. - -Not all artifacts of a Project are included in a sync. Generally we sync the -project's workflows and some of its options. But we do not sync any associated -data, credential values, usage history or AI sessions. - -## Authorization - -Before you use the CLI to fetch anything from the app, you'll need to provide -some authorization. - -The best way to do this is to set an environment variable called -`OPENFN_API_KEY`. Set it to the value of your -[Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). - -:::info Personal Access Tokens - -See -[Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) -for help setting up a token. - -::: - -If you connect to multiple OpenFn projects or apps, you can create a `.env` file -and set any environment variables you need there. The CLI will load this file -and report which keys its using. Values in your `.env` file will be preferred to -those defined in your system. - -You can also pass `--api-key` directly as a flag to most commands. - -:::info - -This guide assumes you want to sync with our hosted SaaS app at -[app.openfn.org](https://app.openfn.org) - -You can sync with a different OpenFn instance by setting the env var -`OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. - -::: - -## Downloading a Project - -To pull a project down from the app to your local machine, run: - -```bash -openfn project pull -``` - -This will create a file in your working directory called -`.projects/main@app.openfn.org.yaml`. - -:::info - -Every Project in the app has a unique identifier, called a UUID, which is used -to reference it. It is a 32 digit number of the form -`a6cc5bdd-b04f-4413-b4b8-132a5115acac` - -You can copy a Project's UUID from the URL by browsing to it in the app. It's -the long string after `projects`. - -For example, the UUID is the bold section in: -{'https://openfn.org/projects/'}{'abc087dd-3963-4260-8d09-ced2e1ff2bb0'}{'/w'} - -::: - -After you've pulled a project for the first time, you don't need to specify the -UUID again. You can use the alias, the id, or leave the identifier blank to use -the currently checked out project. - -The `project pull` command does three things: - -- If you do not have an `openfn.yaml` file, it'll create one -- It will _fetch_ (download) your project from the app and save it into a single - file at `.projects/main@app.openfn.org.yaml` -- It will _checkout_ (expand) that project onto your file system, expanding - workflows and steps to their own files. - -## Understanding Project Structure - -Pulling a project from the app will create a file structure that looks like -this: - -``` -├── openfn.yaml -├── .projects -│ ├── main@app.staging.openfn.org.yaml -└── workflows - ├── my-workflow - │ ├── my-workflow.yaml - │ ├── job.js -``` - -There are three key files to understand here: - -### project.yaml - -The project file saves a copy of the whole state of a project as saved in the -app. If you look inside you'll see the workflows represented as plain text. - -A project file is named like `@.yaml`. The alias is a local name -used to refer to a particular version of the project. The domain comes from the -OpenFn instance the project was downloaded from. - -The project file should not be edited locally as any changes will be dropped on -the next fetch. - -You can fetch as many projects as you like, and each will be saved to its own -project.yaml file. - -The `.projects` folder can and should be committed to source control. - -### workflows - -Having your whole project inside a single file isn't actually a great way to -read or edit workflows. So the CLI can "checkout" or "expand" a project file -onto the file system. - -Checking out is the process of writing each workflow to a workflow.yaml file and -each step to a step.js file. This all lives in the `workflows` directory. - -Here you can edit files as much as you like, and changes will be tracked when -you push/deploy back to the app. - -You can only check out one project at a time. This is actually great for working -with git, because you can checkout two projects on different branches and -compare/merge them against each other directly. - -### openfn.yaml - -This is a top-level configuration file which can mostly be ignored. It is used -by OpenFn tooling to recognise a project root folder. It also holds -configuration options for all local projects, and metadata about the currently -checked out project. - -## Aliases - -Rather than identify a local project with a long-winded UUID or id string, you -can use an _alias_. - -Each local project is saved to a file like `main@app.openfn.org.yaml`, where the -`main` bit is the local alias of a project. - -You can pull a project and set the alias at the same time by running - -```bash -openfn project pull --alias dev -``` - -This will save the project to `dev@app.openfn.org.yaml`. - -To change the alias, you can simply rename the file. Whatever goes before the -`@` will be treated as the alias. - -## Checking Out - -You can checkout a project any time with: - -```bash -openfn project checkout -``` - -This will update your local workflows folder with the target project. - -If a checkout will cause changes to be lost (ie, you've changed a step.js file -but haven't deployed it), you will be warned. Add `--force` to ignore the -change, or run `openfn project clean` to wipe and reset the `workflows` folder. - -Checking out will only modify files managed by the CLI - basically workflow and -step files. If you have other files in the file system (like state files or test -files) they will be untouched. - -## Running workflows in projects - -You can execute any workflow in the checked out project by name: - -```bash -openfn my-workflow -``` - -The CLI will find the workflow in your `workflows` folder and run it. You can -pass state via `-s` and set log levels as per usual. - -When running a workflow by name like this, you get two benefits: - -- **Credentials** are loaded automatically from the credential map in - `openfn.yaml`, so you don't need to pass `--credential-map` -- **Collections** use the server configured in `openfn.yaml`, so you don't need - to pass `--collections-endpoint` or anything. - -## Deploying a Project - -To push your local changes back to the app, run: - -```bash -openfn project deploy -``` - -This will take your currently checked out project and push it to the app. It -will also report what has changed in the local project. - -Before uploading, the CLI fetches the latest version of the project from the app -and checks for **divergence** — meaning it checks whether any of your -locally-changed workflows have also been edited in the app since you last -pulled. If they have, the deploy will fail with an error to prevent you from -accidentally overwriting someone else's work. - -If you want to push anyway, pass `--force`: - -```bash -openfn project deploy --force -``` - -To preview what would change without actually uploading anything, use -`--dry-run`. This will log the final update payload that would be sent to the -app (as a JSON structure). - -You can deploy the checked out project as a new project on the target app by -adding the `--new` flag. This is only available if you have superuser privileges -on the target instance. - -You can also deploy the checked out project to another app project by passing -its alias, id or uuid: - -``` -openfn project deploy main -``` - -If you currently have a development sandbox checked out, this would merge it -straight into the main app project. - -Note that you have to have fetched the target project locally before you can -deploy it. - -## Sandboxes - -The CLI is fully compatible with sandboxes. Treat them like any other project: -fetch them for the first time with their UUID. - -Use the `checkout` command to switch between sandboxes and projects locally. -Remember you can only have a single project checked out at once. The CLI will -warn you if a checkout will cause you to lose any local changes. - -When fetching a sandbox, by default the project alias will be set to the sandbox -name. - -You can merge two projects locally with `openfn project merge`, and deploy the -resulting project to the app (you'll likely have to force push the change). This -is useful for conflict resolution. - -## GitHub - -You can configure a Project to sync automatically to GitHub. This means that -commit to GitHub will automatically deploy changes to an OpenFn Project; and -pressing Save & Sync in the app will commit back to GitHub. - -Under the hood, GitHub Sync uses the CLI's `pull` and `deploy` commands, -triggered by GitHub Actions, to sync your projects. - -Note that by default, GitHub Sync will use the legacy format, with `state.json`, -`project.yaml` and `config.json` files. When setting up a new GitHub Sync, you -can choose to select the v2 format. V2 sync is only suitable for pulling a -single Project per branch to github, as multiple projects will overwrite the -same `workflows/` folder. - -See [Version Control](/documentation/link-to-GitHub) for more details about -GitHub Sync. - -## Cheatsheet - -| Command | Description | -| ---------------------------------------- | ------------------------------------------------------------------------------ | -| `openfn project pull ` | Pull a project from the app for the first time | -| `openfn project pull` | Re-pull the current project | -| `openfn project pull --alias dev` | Pull and set a local alias | -| `openfn project fetch ` | Fetch a project without checking it out | -| `openfn project` | List all local projects in the current working folder | -| `openfn project checkout ` | Switch to a different local project | -| `openfn project deploy` | Deploy checked-out project to the app | -| `openfn project deploy --dry-run` | Try a deploy but skip the upload step | -| `openfn project deploy --force` | Force the checked out project to be uploaded, ignoring any divergence warnings | -| `openfn ` | Run a workflow in the checked-out project | -| `openfn project clean` | Delete the `workflows` folder and all contents, then check out the project | diff --git a/sidebars-main.js b/sidebars-main.js index 6cb20f464cbd..404fc48fe25d 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -124,7 +124,6 @@ module.exports = { items: [ 'build-for-developers/cli-intro', 'build-for-developers/cli-usage', - 'build-for-developers/cli-sync', 'build-for-developers/cli-collections', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', From 1028de02e5452687afafdf038380030ac930a8a8 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 7 May 2026 18:27:34 +0100 Subject: [PATCH 12/18] updates --- .../{portability-v1.md => portability-v3.md} | 2 +- docs/deploy/portability.md | 495 ++++-------------- 2 files changed, 107 insertions(+), 390 deletions(-) rename docs/deploy/{portability-v1.md => portability-v3.md} (99%) diff --git a/docs/deploy/portability-v1.md b/docs/deploy/portability-v3.md similarity index 99% rename from docs/deploy/portability-v1.md rename to docs/deploy/portability-v3.md index 679872d94e2c..51e848e6fcba 100644 --- a/docs/deploy/portability-v1.md +++ b/docs/deploy/portability-v3.md @@ -1,5 +1,5 @@ --- -title: Portability v1 (legacy) +title: Portability v3 (legacy) --- The portability specification allows for the representations of entire workflow diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 54aea57f5eac..fe0d01c0a9a5 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -2,11 +2,30 @@ title: Portability --- -The portability specification allows for the representations of entire workflow -projects "as code"; lets users move between various deployment pathways (such as -cloud, local, hosted); and proposes a globally-applicable way of defining -workflow automation and system integration rules that might be applied across -workflow-engines/integration platforms across the sector. +The Portability Specification is an idea right at the heart of OpenFn Projects. +It is both a technical standard and an ongoing commitment. + +The Portability Specification ensures that code written in an OpenFn application +can be: + +- Deployed to another OpenFn instance (critical for production services running + in-country) +- Executed on a local machine (great news for developers) +- Ejected from OpenFn entirely and executed through a generic JavaScript runtime + +The Portability Specification drives the core functionality of OpenFn Sync, CLI +Deploy, Sandbox merging, and Project export/import from the app. + +:::info Legacy Portability Specifications + +Our commitment to portability hasn't changed within OpenFn's lifetime - but our +approach and implementation of this commitment have taken many forms. + +This document describes the latest Portability Specification, published in +May 2026. For older specifications, see +[Portability Versions](portability-versions) + +::: Nothing about the spec _must_ be specific to OpenFn or any one of our individual products. We envision a future in which software built with Lightning, the @@ -17,20 +36,10 @@ If you're interested in contributing to the specification, reach out to OpenFn via the [community forum](https://community.openfn.org), write to us, or suggest changes by submitting a pull request here. -:::info +## Projects As Code -This document describes v2 of the OpenFn deployment spec, in-line with -[OpenFn Sync](/documentation/sync). - -For the v1 spec (using state, project and config files), see -[Legacy Portability](./portability-v1). - -::: - -## Projects "as code" - -Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. +A core tenant of OpenFn Projects is that they can be represented as code, on a +file system or a git branch. This improves the OpenFn developer experience by: @@ -39,405 +48,113 @@ This improves the OpenFn developer experience by: 3. Allowing users to port existing projects between different instances (i.e., deployments) of Lightning. -### Directory structure - -Many users keep OpenFn projects in git repositories, and this is a common -structure: - -``` -myProject/ -├── workflow-a/ -│ ├── job-1.js -│ ├── job-2.js -│ └── job-3.js -├── workflow-b/ -│ └── job-4.js -├── project.yaml -├── projectState.json -└── config.json -``` - -:::info Directory Structure - -There are commonly used 3 directory structure for OpenFn projects namely: -standard, production & test, and monorepo. To learn more, please see the OpenFn -[GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). - -::: - -### The project **_spec_** - -The project specification (or "spec") is often saved as a `project.yaml` file. -While most of the spec is written inline, many developers prefer to track their -job bodies in separate `.js` files and they then reference them with a relative -path. - -```yaml -name: openhie-project -description: Some sample -credentials: - jane-smith@test.com-HAPI-FHIR: - owner: jane-smith@test.com - name: HAPI FHIR -workflows: - OpenHIE-Workflow: - name: OpenHIE Workflow - jobs: - FHIR-standard-Data-with-change: - name: FHIR-standard-Data-with-change - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: - path: ./jobs/my-fancy-script.js - - Send-to-OpenHIM-to-route-to-SHR: - name: Send-to-OpenHIM-to-route-to-SHR - adaptor: '@openfn/language-http@latest' - enabled: true - credential: jane-smith@test.com-HAPI-FHIR - body: | - fn(state => { - console.log("hello github integration") - return state - }); - - Notify-CHW-upload-successful: - name: Notify-CHW-upload-successful - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: fn(state => state); - - Notify-CHW-upload-failed: - name: Notify-CHW-upload-failed - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: - path: ./jobs/notify-failure.js - - triggers: - webhook: - type: webhook - edges: - webhook->FHIR-standard-Data-with-change: - source_trigger: webhook - target_job: FHIR-standard-Data-with-change - condition: always - FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: - source_job: FHIR-standard-Data-with-change - target_job: Send-to-OpenHIM-to-route-to-SHR - condition: on_job_success - Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: - source_job: Send-to-OpenHIM-to-route-to-SHR - target_job: Notify-CHW-upload-successful - condition: on_job_success - Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: - source_job: Send-to-OpenHIM-to-route-to-SHR - target_job: Notify-CHW-upload-failed - condition: on_job_failure -``` - -In this spec, you can see the different ways of defining a job's body: - -1. Inline body: Used in the `FHIR-standard-Data-with-change` and - `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the - YAML file. - -2. External file reference: Used in both `Notify-CHW-upload-successful` and - `Notify-CHW-upload-failed` jobs. The body is stored in separate files, - referenced by the path key. This allows for better organization of complex - job logic. - -When using file paths: - -- Paths are relative to the location of the `project.yaml` file. -- Ensure that the referenced files exist and contain valid job body code. -- This method is particularly useful for complex jobs or when you want to reuse - job bodies across different projects. - -### The project **_state_** - -The project state is a representation of a particular project as _on a specific -Lightning instance_. It is often saved as `projectState.json` and contains UUIDs -for resources on a particular Lightning deployment. - -```json -{ - "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", - "name": "openhie-project", - "description": null, - "inserted_at": "2023-08-25T08:57:31", - "updated_at": "2023-08-25T08:57:31", - "scheduled_deletion": null, - "requires_mfa": false, - "project_credentials": { - "jane-smith@test.com-HAPI-FHIR": { - "id": "25f48989-d349-4eb8-99c3-923ebba5b116", - "name": "HAPI FHIR", - "owner": "jane-smith@test.com" - } - }, - "workflows": { - "OpenHIE-Workflow": { - "id": "27ae2937-0959-48b8-a597-b1646aae8c14", - "name": "OpenHIE Workflow", - "jobs": { - "Transform-data-to-FHIR-standard": { - "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", - "delete": true - }, - "Send-to-OpenHIM-to-route-to-SHR": { - "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "name": "Send-to-OpenHIM-to-route-to-SHR", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "Notify-CHW-upload-successful": { - "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", - "name": "Notify-CHW-upload-successful", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "Notify-CHW-upload-failed": { - "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", - "name": "Notify-CHW-upload-failed", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "FHIR-standard-Data": { - "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", - "delete": true - }, - "FHIR-standard-Data-with-change": { - "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", - "name": "FHIR-standard-Data-with-change", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - } - }, - "triggers": { - "webhook": { - "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", - "type": "webhook" - } - }, - "edges": { - "webhook->Transform-data-to-FHIR-standard": { - "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", - "delete": true - }, - "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { - "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", - "delete": true - }, - "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { - "id": "26c12f7f-7806-4008-87cd-6747998f95f4", - "condition": "on_job_success", - "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "source_trigger_id": null, - "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" - }, - "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { - "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", - "condition": "on_job_failure", - "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "source_trigger_id": null, - "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" - }, - "webhook->FHIR-standard-Data": { - "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", - "delete": true - }, - "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { - "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", - "delete": true - }, - "webhook->FHIR-standard-Data-with-change": { - "id": "75e7f7d8-274b-410d-9600-730bbd535229", - "condition": "always", - "source_job_id": null, - "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", - "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" - }, - "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { - "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", - "condition": "on_job_success", - "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", - "source_trigger_id": null, - "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" - } - } - } - } -} -``` +## Project Spec -## Using the CLI interact with projects +:::warning TODO -The project spec and project state can be used for a variety of reasons, e.g. -one could generate the state and spec as backups of the project or one could -generate these files and use them for auditing and record keeping, etc. The -OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with -commands that can be used to pull project configurations down from a running -Lightning server, and to deploy or push updates to existing projects on a -Lightning server. To learn more about automated version control via pull and -deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) -docs. - -:::info Don't have the CLI yet? - -Install it by running `npm install -g @openfn/cli` +project file? Project text? Project source? Oh I like source ::: -Before using the CLI, configure it either by passing in environment variables: - -``` -OPENFN_ENDPOINT=https://app.openfn.org -OPENFN_API_KEY=yourSecretApiToken -``` - -Or through a `config.json` file: - -```json -{ - // Required, can be overridden or set with `OPENFN_API_KEY` env var - "apiKey": "***", +The unit of portability - the thing that encodes a Project and allows it to be +shared, synced, deployed and edited - is called a Project Spec. - // Optional: can be set using the -p, defaults to project.yaml - "specPath": "project.yaml", +It is a structured artifact which defines a set of workflows, and for each +workflow, its, its core configuration, and the sequence of steps which it +executes. We usually represent this structure as YAML, because it's convenient +for humans and machines, but it can be represented in any text format. - // Optional: can be set using -s, defaults to .state.json - "statePath": ".state.json", +With a copy of a project spec, users can: - // Optional: defaults to OpenFn.org's API, can be overridden or set with - // `OPENFN_ENDPOINT` env var - "endpoint": "https://app.openfn.org" -} -``` - -More details on the CLI can be found -[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). +- Import a project into an OpenFn app instance +- Execute workflows locally with the CLI +- Deploy a project to an OpenFn app instance +- Merge sandbox projects locally -### `openfn pull` to generate spec & state +Keys are regularly added to this structure as new features are introduced. We +expect and ensure that these keys are supported in all applications of the spec. -To generate the spec and state files for an existing project, use: - -```sh -openfn pull {YOUR-PROJECT-UUID} -c ./config.json -``` +Project specs can be exported from the app via the Settings page. -This command will save (or overwrite) a project spec and state file based on the -path you've set in your configuration. +Workflows can also be interchanged independently, using the same spec. So you +can import a Workflow to an existing project, or execute it locally without +cloning a whole project. -### `openfn deploy` to create new projects +## Spec Example -To deploy a new project to a Lightning instance from a project spec (without a -project state) file use: +Here is an example Project spec in YAML format: -```sh -openfn deploy -c config.json +```yaml +id: joe-sync-1 +name: Joe Sync 1 +description: Duplication of staging joe-sync-1 +collections: + - joestuff +credentials: + - name: http test + owner: editor@openfn.org +options: + allow_support_access: false + requires_mfa: false + retention_policy: retain_all +workflows: + - name: A + steps: + - id: a + name: a + expression: fn(s => s) + adaptor: '@openfn/language-http@7.2.9' + configuration: editor@openfn.org|http test + - id: webhook + type: webhook + enabled: true + next: + a: + disabled: false + condition: always + id: a + start: webhook ``` -### `openfn deploy` to update existing projects - -With a valid project state defined in your `config.json`, the same -`openfn deploy` command will beam up your changes as described by a difference -between your project spec and what's found on the server. - -```sh -openfn deploy -c config.json -Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. -Project found. -[CLI] ♦ Changes: - { - workflows: [ - { - jobs: [ - { -- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" -+ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" - } - ... - ... - ... - ] - } - ] - } - -? Deploy? yes -[CLI] ♦ Deployed. -``` +The latest schema for a project spec file is defined in TypeScript +[here](https://github.com/OpenFn/kit/blob/5e4d65af25a6854886c15294ed4cf17f93ecbc19/packages/lexicon/portability.d.ts) -## Getting Help with the cli +## Linked Resources -The cli package comes with an inbuilt `help`. Adding `--help` to a command such -as `openfn deploy --help` will result in a help message describing the command -and the options available when using this command. See an example below +:::warning TODO -```sh -openfn deploy --help -openfn deploy +I don't really want to talk about state here - but I do want to touch on the +idea that the spec is uncoupled from an instance, and that we have these "linked +resources" which on deploy get assigned UUIDs -Deploy a project's config to a remote Lightning instance +But the new statefile, main@app.openfn.org.yaml, is not part of the portability +spec! It's just an artefact used by the CLI to track and instance. -Options: - --version Show version number [boolean] - --help Show help [boolean] - -c, --config, --config-path The location of your config file [default: "./.config.json"] - --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] - --describe Downloads the project yaml from the specified instance [boolean] - -l, --log Set the log level [string] - --log-json Output all logs as JSON objects [boolean] - -p, --project-path The location of your project.yaml file [string] - -s, --state-path Path to the state file -``` +If the SPEC is standard for interopability which might go beyond openfn, the +STATEFILE isa proprietary artefact of sync which is 100% coupled to the app -## Troubleshooting +::: -This section covers solutions to some errors you might come across when using -OpenFn pull or deploy in your projects. +Project State links a project to related artifacts which live on a specific +instance of the app - like a user or a credential. -### Extraneous Workflow ID +The Project Spec does not link directly to those resources. Rather, it +references those resources by an identifier string. -#### Description +When syncing or deploying a project, the hosting app will attempt to reference +an artifact with the same identifier within the project's scope, and connect it. +That connection is local to that instance and usually requires a UUID reference. +This connection is not portable. -This error occurs when you run `openfn deploy` and there is a mismatch between -between IDs of workflows in your projectSpec and your OpenFn instance. When this -occurs, the error will be written out in an error object as shown below: +:::warning TODO -``` -[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: -{ - "errors": { - "workflows": { - "1-ingest-messages": { - "base": [ - "extraneous parameters: workflow_id" - ] - }, - "2-calculate-indicators": { - "base": [ - "extraneous parameters: workflow_id" - ] - } - } - } -``` +I want to neatly reference users of the portability spec. -#### Solution +Like "see CLI Sync" to read more about how portability works in practice. -Run `openfn pull` to update your local instance and keep IDs in sync, -incorporate your changes and run `openfn deploy` again. +::: -## Other Versions +## Execution -- [Portability Spec v2](portability-versions#v2) -- [Portability Spec v1](portability-versions#v1) +TODO: touch on executing workflows with CLI, and also compiling a workflow and +running that with native node. From fb7fa4ae8b629e6bc4382412662bf6e376b87087 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Fri, 8 May 2026 17:50:35 +0100 Subject: [PATCH 13/18] update example --- docs/deploy/portability.md | 68 +++++++++++++++++++++++++++++--------- 1 file changed, 52 insertions(+), 16 deletions(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index fe0d01c0a9a5..6951301e1e43 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -85,35 +85,71 @@ cloning a whole project. Here is an example Project spec in YAML format: ```yaml -id: joe-sync-1 -name: Joe Sync 1 -description: Duplication of staging joe-sync-1 +id: portability-example +name: Portability Example collections: - - joestuff + - cache credentials: - - name: http test + - name: local login owner: editor@openfn.org -options: - allow_support_access: false - requires_mfa: false - retention_policy: retain_all workflows: - - name: A + - name: Event-based workflow steps: - - id: a - name: a + - id: transform-data + name: Transform data expression: fn(s => s) - adaptor: '@openfn/language-http@7.2.9' - configuration: editor@openfn.org|http test + adaptor: '@openfn/language-common@latest' - id: webhook type: webhook + webhook_reply: before_start enabled: true next: - a: + transform-data: disabled: false condition: always - id: a + id: event-based-workflow start: webhook + - name: Scheduled workflow + steps: + - id: common + name: Common + expression: fn(s => s) + adaptor: '@openfn/language-common@3.3.1' + - id: cron + type: cron + enabled: true + cron_expression: 00 00 * * 1-5 + cron_cursor_job_id: fb95ea89-17d2-4773-823c-09770317aaed + next: + get-data: + disabled: false + condition: always + - id: get-data + name: Get data + expression: fn(s => s) + adaptor: '@openfn/language-http@latest' + configuration: editor@openfn.org|local login + next: + throw-error: + disabled: false + condition: on_job_failure + common: + disabled: false + condition: '!state.error' + label: sometimes + never: + disabled: true + condition: on_job_success + - id: never + name: never + expression: fn(s => s) + adaptor: '@openfn/language-http@7.2.10' + - id: throw-error + name: throw error + expression: fn(s => s) + adaptor: '@openfn/language-common@3.3.1' + id: scheduled-workflow + start: cron ``` The latest schema for a project spec file is defined in TypeScript From 9018225d3e3a566608d11bb3631cc37f13e7f9e4 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Sun, 10 May 2026 12:21:08 +0100 Subject: [PATCH 14/18] tidy and update docs --- docs/deploy/portability.md | 83 +++++++++++++++----------------------- 1 file changed, 33 insertions(+), 50 deletions(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 6951301e1e43..f5efbc3da1ea 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -3,18 +3,17 @@ title: Portability --- The Portability Specification is an idea right at the heart of OpenFn Projects. -It is both a technical standard and an ongoing commitment. - -The Portability Specification ensures that code written in an OpenFn application -can be: +It is both a technical standard and an ongoing commitment. It ensures that code +written in an OpenFn application can be: - Deployed to another OpenFn instance (critical for production services running in-country) -- Executed on a local machine (great news for developers) +- Executed on a local machine (great news for developers building workflows or + adaptors) - Ejected from OpenFn entirely and executed through a generic JavaScript runtime -The Portability Specification drives the core functionality of OpenFn Sync, CLI -Deploy, Sandbox merging, and Project export/import from the app. +This manifesto drives the core functionality of OpenFn Sync, CLI Deploy, Sandbox +merging, and Project export/import from the app. :::info Legacy Portability Specifications @@ -50,19 +49,15 @@ This improves the OpenFn developer experience by: ## Project Spec -:::warning TODO - -project file? Project text? Project source? Oh I like source - -::: - The unit of portability - the thing that encodes a Project and allows it to be -shared, synced, deployed and edited - is called a Project Spec. +shared, synced, deployed and edited - is called a Project Spec. It is an +abstract definition of a project, a blueprint which can be deployed to many +places. -It is a structured artifact which defines a set of workflows, and for each -workflow, its, its core configuration, and the sequence of steps which it -executes. We usually represent this structure as YAML, because it's convenient -for humans and machines, but it can be represented in any text format. +This structure defines a set of workflows, and for each workflow, its, its core +configuration, and the sequence of steps which it executes. We usually represent +this structure as YAML, because it's convenient for humans and machines, but it +can be represented in any text format. With a copy of a project spec, users can: @@ -155,42 +150,30 @@ workflows: The latest schema for a project spec file is defined in TypeScript [here](https://github.com/OpenFn/kit/blob/5e4d65af25a6854886c15294ed4cf17f93ecbc19/packages/lexicon/portability.d.ts) -## Linked Resources - -:::warning TODO - -I don't really want to talk about state here - but I do want to touch on the -idea that the spec is uncoupled from an instance, and that we have these "linked -resources" which on deploy get assigned UUIDs +## Syncing Projects -But the new statefile, main@app.openfn.org.yaml, is not part of the portability -spec! It's just an artefact used by the CLI to track and instance. +For more details about how a project can be deployed, executed, pulled and +edited, see our extensive documentation on [CLI Sync](/documentation/sync). -If the SPEC is standard for interopability which might go beyond openfn, the -STATEFILE isa proprietary artefact of sync which is 100% coupled to the app - -::: - -Project State links a project to related artifacts which live on a specific -instance of the app - like a user or a credential. - -The Project Spec does not link directly to those resources. Rather, it -references those resources by an identifier string. - -When syncing or deploying a project, the hosting app will attempt to reference -an artifact with the same identifier within the project's scope, and connect it. -That connection is local to that instance and usually requires a UUID reference. -This connection is not portable. - -:::warning TODO +## Linked Resources -I want to neatly reference users of the portability spec. +While we designed Projects with portabality in mind - some features are +intrinsically NOT portable. -Like "see CLI Sync" to read more about how portability works in practice. +For example, credentials contain highly secure tokens, which by design and +nature should be very hard to extract from the OpenFn platform. So credentials +aren't really portable. When exporting a project, sensitive credentials should +be included in that exported, plain-text document. -::: +Similarly, collections are a feature which is tied closely to a specific +deployment of an OpenFn platform. The data of collections is not covered by the +portability spec (although with the right permissions data can be synced between +collections). -## Execution +These kinds of non-portable resources are not part of a project, but they are +LINKED to a project. -TODO: touch on executing workflows with CLI, and also compiling a workflow and -running that with native node. +Usually resources are linked by name. A credential and collections just declares +a dependency on a thing with a given name, which must be resolved at runtime. +The CLI has tools to do that, and when deploying to a target instance the +instance may need to be pre-configured to have matching resources. From effa3af372c2484a997c75a7cd13a0833d18e787 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Sun, 10 May 2026 12:23:35 +0100 Subject: [PATCH 15/18] fix legacy --- docs/deploy/portability-versions.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/deploy/portability-versions.md b/docs/deploy/portability-versions.md index 8c7d5cd4dd24..a4eb8bc56014 100644 --- a/docs/deploy/portability-versions.md +++ b/docs/deploy/portability-versions.md @@ -2,12 +2,20 @@ title: Versions of the Portability Proposal --- -OpenFn is currently designing a portable project configuration schema that can -be used to import or export projects between OpenFn/platform and OpenFn/engine. +Our commitment to portability hasn't changed within OpenFn's lifetime - but our +approach and implementation of this commitment have taken many forms. -## [Current](/documentation/deploy/portability#the-project-spec) +This document provides a reference to older versions of specification. -[See the current specification here.](/documentation/deploy/portability#the-project-spec) +## v3 + +The v3 standard was created for the v2 platform and is linked to the Lightning +project. + +V3 uses the legacy CLI deploy commands and protocols. It is still full supported +by the app and CLI, but is being phased out as of May 2026. + +[See the v3 specification here](/documentation/deploy/portability-v3) ## v2 From 6ef38afbc8a9c0173bfa3a32dcaf9f43565a7962 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Sun, 10 May 2026 12:27:43 +0100 Subject: [PATCH 16/18] typos --- docs/deploy/options.md | 4 ++-- docs/deploy/portability-versions.md | 4 ++-- docs/deploy/portability.md | 21 +++++++++++++-------- docs/deploy/requirements.md | 7 ++++--- 4 files changed, 21 insertions(+), 15 deletions(-) diff --git a/docs/deploy/options.md b/docs/deploy/options.md index bef0b3c53e2a..426fe8d172b0 100644 --- a/docs/deploy/options.md +++ b/docs/deploy/options.md @@ -46,11 +46,11 @@ local/government-managed servers, you might: preparations**: 1. Assess the **value of the solution** itself—is it solving the problems you thought it would? - 2. Asses your **data residency requirements**—do you need to run this + 2. Assess your **data residency requirements**—do you need to run this solution in country? 3. Assess your team’s technical **DevOps capacity**—how are other local deployments of DPGs going? - 4. Assess your countries computing, storage, and networking + 4. Assess your country's computing, storage, and networking infrastructure—what options\* are available for servers and network connectivity? 5. Determine if a **"zero-persistence"** cloud solution or a **locally diff --git a/docs/deploy/portability-versions.md b/docs/deploy/portability-versions.md index a4eb8bc56014..4a3bf7867526 100644 --- a/docs/deploy/portability-versions.md +++ b/docs/deploy/portability-versions.md @@ -5,14 +5,14 @@ title: Versions of the Portability Proposal Our commitment to portability hasn't changed within OpenFn's lifetime - but our approach and implementation of this commitment have taken many forms. -This document provides a reference to older versions of specification. +This document provides a reference to older versions of the specification. ## v3 The v3 standard was created for the v2 platform and is linked to the Lightning project. -V3 uses the legacy CLI deploy commands and protocols. It is still full supported +V3 uses the legacy CLI deploy commands and protocols. It is still fully supported by the app and CLI, but is being phased out as of May 2026. [See the v3 specification here](/documentation/deploy/portability-v3) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index f5efbc3da1ea..9443312abdde 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -37,7 +37,7 @@ changes by submitting a pull request here. ## Projects As Code -A core tenant of OpenFn Projects is that they can be represented as code, on a +A core tenet of OpenFn Projects is that they can be represented as code, on a file system or a git branch. This improves the OpenFn developer experience by: @@ -54,7 +54,7 @@ shared, synced, deployed and edited - is called a Project Spec. It is an abstract definition of a project, a blueprint which can be deployed to many places. -This structure defines a set of workflows, and for each workflow, its, its core +This structure defines a set of workflows, and for each workflow, its core configuration, and the sequence of steps which it executes. We usually represent this structure as YAML, because it's convenient for humans and machines, but it can be represented in any text format. @@ -157,13 +157,13 @@ edited, see our extensive documentation on [CLI Sync](/documentation/sync). ## Linked Resources -While we designed Projects with portabality in mind - some features are +While we designed Projects with portability in mind - some features are intrinsically NOT portable. For example, credentials contain highly secure tokens, which by design and nature should be very hard to extract from the OpenFn platform. So credentials aren't really portable. When exporting a project, sensitive credentials should -be included in that exported, plain-text document. +not be included in that exported, plain-text document. Similarly, collections are a feature which is tied closely to a specific deployment of an OpenFn platform. The data of collections is not covered by the @@ -173,7 +173,12 @@ collections). These kinds of non-portable resources are not part of a project, but they are LINKED to a project. -Usually resources are linked by name. A credential and collections just declares -a dependency on a thing with a given name, which must be resolved at runtime. -The CLI has tools to do that, and when deploying to a target instance the -instance may need to be pre-configured to have matching resources. +Usually resources are linked by name. Credentials and collections just declare a +dependency on a thing with a given name, which must be resolved at runtime. The +CLI has tools to do that, and when deploying to a target instance the instance +may need to be pre-configured to have matching resources. + +## Legacy Portability Specifications + +For older versions of our approach, see +[Portability Versions](portability-versions) diff --git a/docs/deploy/requirements.md b/docs/deploy/requirements.md index 3c24072d9afc..dee382b455f9 100644 --- a/docs/deploy/requirements.md +++ b/docs/deploy/requirements.md @@ -63,7 +63,7 @@ independently of the Javascript worker node pool & pods. the following specs will help prevent unwanted downtime. 1. **GKE requests:** cpu@ "500m", memory@ "1024Mi" 2. **GKE limits:** memory@ "2560Mi" -2. For a simple non-Kubernetes/HA deployments, the minimum recommended machines +2. For simple non-Kubernetes/HA deployments, the minimum recommended machines are: - **Application machine:** 2 vCPU (roughly a single core of a 2.6 GHz Intel Xeon E5) with 3.75 GB memory and 15 gb of storage for the application @@ -75,8 +75,9 @@ independently of the Javascript worker node pool & pods. days (if any) of message data you’d like to store on the app itself and cannot be determined without estimates for message/run throughput. If scaling physical storage is not difficult for your particular deployment, - start at 40gb. 3. A Postgres (at least v14.2) instance (as we run this on a - _separate server_) from the application for greater stability. + start at 40gb. + 1. A Postgres (at least v14.2) instance, run on a _separate server_ from + the application for greater stability. 3. If both the application and database are hosted on the same machine (which is not recommended) that machine should have roughly the sum of the requirements above. From 61ecbee7b54321a0382d8601ec4177467baca68a Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Sun, 10 May 2026 12:28:27 +0100 Subject: [PATCH 17/18] update link --- docs/deploy/portability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 9443312abdde..2a364b6da56f 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -148,7 +148,7 @@ workflows: ``` The latest schema for a project spec file is defined in TypeScript -[here](https://github.com/OpenFn/kit/blob/5e4d65af25a6854886c15294ed4cf17f93ecbc19/packages/lexicon/portability.d.ts) +[here](https://github.com/OpenFn/kit/blob/main/packages/lexicon/portability.d.ts) ## Syncing Projects From 583ed0ad378672cb57b038ba75114f4020d705fc Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 11 May 2026 09:51:25 +0100 Subject: [PATCH 18/18] fix uuid --- docs/deploy/portability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 2a364b6da56f..275e0a6f12eb 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -114,7 +114,7 @@ workflows: type: cron enabled: true cron_expression: 00 00 * * 1-5 - cron_cursor_job_id: fb95ea89-17d2-4773-823c-09770317aaed + cron_cursor_job_id: get-data next: get-data: disabled: false