Portability Spec

[josephjclark]

Introduce a formal portability layer into the project's typings.

The idea is to use TypeScript to define the schema of the abstract project yaml: a file which must be imported and exported by Lightning, executed and deployed with the CLI. Workflow yaml is a subset of this project.yaml.

This is forcing a health healthy, wholesome review of key types across in the codebase.

The key bit is in https://github.com/OpenFn/kit/blob/5e4d65af25a6854886c15294ed4cf17f93ecbc19/packages/lexicon/portability.d.ts

Implementation Details

A major concern of this work is to introduce a better separation between:

• the abstract project.yaml which is the unit of portability
• The serialized project state file (which we write to yaml), including UUIDs
• Internal schema keys used by the core runtime to execute jobs

This PR does not, and is not attempting to, fix every single type discrepancy. There are TODOs in the code. I just want to get the architecture right.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our
Responsible AI Policy

[josephjclark]

TODO: ensure that adaptors is an implemention detail on the runtime, an adaptor is the name of the key on the spec

[taylordowns2000]

questions about the interface between this spec and lightning specific stuff around triggers

[taylordowns2000]

how do we deal with things like cron_cursor_job_id

stuff like that, or webhook_reply, or all the kafka stuff topics, initial_offset_reset_policy, ...otherStuff

a. all just flat as keys at the trigger root?
b. all in a kafka: ... or cron: ... or webhook: ... configuration block?
c. all in a lightning: ... configuration block?

[josephjclark]

Voting a) for now!

In the state file the CLI uses (which isn't part of the portability spec IMO) we store app specific stuff in a key called openfn. Which is basically your lightning key. But I'm not really sure that's working out too well tbh - it's a bit inconsistent which keys are there and which are in the main object.

But that doesn't matter too much in the statefile. We do need to have a clean story in the spec and the story is (for now): trigger nodes can hold any key as a configuration option

[taylordowns2000]

perfect. this is how i've built the exporter for lightning (as in: https://github.com/OpenFn/lightning/pull/4722/changes#diff-aae828fd7e4b46b9e5af896155701ad6162c89e9907ec59d416858102581f7e5)