Runtime and Compiler v2

.changeset/README.md

@@ -0,0 +1,8 @@
+# Changesets
+
+Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
+with multi-package repos, or single-package repos to help you version and publish your code. You can
+find the full documentation for it [in our repository](https://github.com/changesets/changesets)
+
+We have a quick list of common questions to get you started engaging with this project in
+[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)

.changeset/config.json

@@ -0,0 +1,11 @@
+{
+  "$schema": "https://unpkg.com/@changesets/config@2.1.1/schema.json",
+  "changelog": "@changesets/cli/changelog",
+  "commit": false,
+  "fixed": [],
+  "linked": [],
+  "access": "public",
+  "baseBranch": "main",
+  "updateInternalDependencies": "patch",
+  "ignore": []
+}

.changeset/eight-donuts-repeat.md

@@ -0,0 +1,5 @@
+---
+'@openfn/logger': patch
+---
+
+Added a confirm utility

.changeset/gold-glasses-shave.md

@@ -0,0 +1,6 @@
+---
+'@openfn/cli': patch
+'@openfn/runtime': patch
+---
+
+Support auto-install of modules

.changeset/great-seas-study.md

@@ -0,0 +1,5 @@
+---
+'@openfn/runtime': patch
+---
+
+Fix default repo location typo

.changeset/sour-points-tie.md

@@ -0,0 +1,5 @@
+---
+'@openfn/describe-package': patch
+---
+
+Include worker bundle in package

.changeset/thick-otters-add.md

@@ -0,0 +1,5 @@
+---
+'@openfn/logger': patch
+---
+
+Export a default logger

.circleci/config.yml

@@ -9,14 +9,14 @@ jobs:
     # Specify the execution environment. You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub.
     # See: https://circleci.com/docs/2.0/configuration-reference/#docker-machine-macos-windows-executor
     docker:
-      - image: cimg/node:16.15
+      - image: cimg/node:18.12
     resource_class: medium
     # Add steps to the job
     # See: https://circleci.com/docs/2.0/configuration-reference/#steps
     steps:
       - run:
           name: Install pnpm
-          command: corepack enable && corepack prepare pnpm@7.5.1 --activate
+          command: corepack enable && corepack prepare pnpm@7.14.2 --activate
       - checkout
       - restore_cache:
           # See the configuration reference documentation for more details on using restore_cache and save_cache steps
@@ -30,6 +30,9 @@ jobs:
           key: deps-v1-{{ .Branch }}-{{checksum "pnpm-lock.yaml"}}
           paths:
             - node_modules
+      - run:
+          name: Build packages
+          command: pnpm build
       - run:
           name: Run Tests
           command: pnpm test

.gitignore

@@ -5,6 +5,7 @@ node_modules
 public/
 !public/index.html
 tsconfig.tsbuildinfo
+ts.cache
 
 # Build tools
 .rollup.cache

.prettierignore

@@ -0,0 +1,2 @@
+node_modules/
+*.md

.prettierrc

@@ -1,5 +1,6 @@
 {
   "tabWidth": 2,
   "useTabs": false,
-  "printWidth": 80
+  "printWidth": 80,
+  "singleQuote": true
 }

.tool-versions

@@ -1 +1 @@
-nodejs 18.8.0
+nodejs 18.12.1

README.md

@@ -1,42 +1,129 @@
-Lightning ProjectSpace
+OpenFn Kit
 ======================
-
 [![CircleCI](https://dl.circleci.com/status-badge/img/gh/OpenFn/kit/tree/main.svg?style=shield)](https://dl.circleci.com/status-badge/redirect/gh/OpenFn/kit/tree/main)
 
+**Kit** _noun_ _/kɪt/_
+
+A set of articles or equipment needed for a specific purpose.
+
+1. _a football kit_
+1. _the next-generation openfn data integration kit_
+
+---
+
+This repo contains runtime, tooling, libraries and components to support the next generation core openfn data integration pipeline.
+
+It is a kitbag of Javascript-based components to support Lightning.
+
+## Prerequisities
+
+* [asdf](https://github.com/asdf-vm/asdf)
+* [pnpm](https://pnpm.io/installation)
+
+We use [asdf](https://github.com/asdf-vm/asdf) to configure our local
+environments and ensure consistency of versions.
+
+You should install asdf and the [NodeJs](https://github.com/asdf-vm/asdf-nodejs) plugin.
+
+We use [`pnpm`](https://pnpm.io/installation), a fast, disk space efficient package manager, to handle node dependencies within the repo.
+
 ## Installing
 
-- Install [`pnpm`](https://pnpm.io/installation)
-- Run `pnpm run setup`
-- Run `pnpm run build`
+- `$ pnpm run setup`
+- `$ pnpm build`
 
-## Packages
+## Running Tests
 
-- [`@openfn/compiler`](packages/compiler)  
-- [`@openfn/workflow-diagram`](packages/workflow-diagram)  
+```
+pnpm run test
+```
+
+# Development Guide
+
+Thanks for being here! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock ❤️
+
+## Releases & Changesets
 
-## Examples
+We use changesets to manage releases: [`github.com/changesets`](https://github.com/changesets/changesets)
 
-The example apps serve to illustrate how these packages can be used, and also
-for development, any changes detected in the dependencies will trigger a rebuild in the example.
+A changeset is a description of batch of changes, coupled with semver information.
 
-**ProjectSpace Flow**
+### Adding a change
+
+When submitting a PR against this repo, include a changeset to describe your work.
 
 ```
-pnpm run -C examples/flow start
+pnpm changeset
 ```
 
-**Compiler Worker**
+For example changeset notes, look in the `.changesets` folder.
+
+### Releasing
+
+To relase to npm:
 
+1) Update versions
 ```
-pnpm run -C examples/compiler-worker start
+pnpm changeset version
 ```
 
-## Running Tests
+This will automatically update the version numbers of the affected packages.
+
+Commit the changes:
 
 ```
-pnpm run test
+git commit -am "Updated changeset version"
 ```
 
+2) Rebuild
+```
+pnpm install
+pnpm build
+```
+
+3) Test
+
+Build the test bundle:
+```
+$ pnpm clean:local
+$ pnpm pack:local
+```
+
+Install using command reported by pack:local (`npm install -g dist/openfn-cli-<version>-local.tgz`)
+
+Sanity check the version and ensure it works:
+```
+$ openfn --version
+$ openfn test
+```
+
+3) Publish
+
+```
+pnpm changeset publish --otp <OTP>
+```
+
+## Testing the built package
+
+You can test the built CLI package to ensure it works before publishing it.
+
+The `build/pack-local` script is an overly complicated solution which:
+* Packs all the openfn packages
+* Updates the dependencies to use peer packages in dist, rather than module names
+* Creates `openfn-cli-<version>-local.tgz` which you can install globally.
+
+To run the test:
+
+```
+$ pnpm build
+$ pnpm clean:local
+$ pnpm pack:local
+```
+
+Run the install command as printed in your shell - something like `npm -g dist/openfn-cli-local.tgz`
+
+You can run `openfn test` to exercise the runtime and compiler.
+
 ## Documentation
 
 For information on the history of the OpenFn internals and ideas for the future

ava.config.js

@@ -0,0 +1,19 @@
+module.exports = {
+  extensions: {
+    ts: "module"
+  },
+
+  environmentVariables: {
+    "TS_NODE_TRANSPILE_ONLY": "true"
+  },
+
+  nodeArguments: [
+    "--loader=ts-node/esm",
+    "--no-warnings", // Disable experimental module warnings
+    "--experimental-vm-modules"
+  ],
+
+  files: [
+    "test/**/*test.ts"
+  ]
+}

build/pack-local.js

@@ -0,0 +1,101 @@
+/**
+ * This will load each of the built .tgzs, extract them in-memory,
+ * update @openfn dependencies in package.json to use absolute local paths,
+ * And write back to disk.
+ */
+const path = require('node:path')
+const fs = require('node:fs');
+const { createGzip } = require('node:zlib');
+const tarStream = require('tar-stream');
+const gunzip = require('gunzip-maybe');
+
+async function findPackages() {
+  return new Promise((resolve) => {
+    fs.readdir(path.resolve('dist'), { encoding: 'utf8' }, (err, files) => {
+      resolve(files.filter(p => p.endsWith('tgz')));
+    });
+  });
+}
+
+const getLocalTarballName = (packagePath) => packagePath.replace('.tgz', '-local.tgz');
+
+function processPackageJSON(stream, packageMap, pack) {
+  return new Promise(resolve => {
+    const data = [];
+    stream.on('data', c => data.push(c))
+    stream.on('end', () => {
+      const buf = Buffer.concat(data)
+
+      const pkg = JSON.parse(buf.toString('utf8'));
+      for (const dep in pkg.dependencies) {
+        if (packageMap[dep]) {
+          console.log(`Mapping ${dep} to ${packageMap[dep]}`)
+          pkg.dependencies[dep] = getLocalTarballName(packageMap[dep]);
+        }
+      }
+      pack.entry({ name: 'package/package.json' }, JSON.stringify(pkg, null, 2), resolve)
+    });
+  });
+}
+
+function updatePkg(packageMap, filename) {
+  const pkgPath = `dist/${filename}`;
+  console.log(' - Updating package', pkgPath)
+
+  // The packer contains the new (gzipped) tarball
+  const pack = tarStream.pack();
+  pack.pipe(createGzip());
+
+  return new Promise((resolve) => {
+    // The extractor streams the old tarball
+    var extract = tarStream.extract();
+    extract.on('entry', (header, stream, next) => {
+      if (header.name === 'package/package.json') {
+        processPackageJSON(stream, packageMap, pack).then(next)
+      }
+      else {
+        stream.pipe(pack.entry(header, next));
+      }
+    });
+
+    // Pipe to a -local file name
+    // Reading and writing to the same tarball seems to cause problems, funnily enough
+    const out = fs.createWriteStream(getLocalTarballName(pkgPath))
+    // Note that we have to start piping to the output stream immediately,
+    // otherwise we get backpressure fails on the pack stream
+    pack.pipe(out);
+
+    fs.createReadStream(pkgPath)
+      .pipe(gunzip())
+      .pipe(extract)
+
+    extract.on('finish', () => {
+      pack.finalize()
+      resolve();
+    });
+  });   
+}
+
+// Map the tgz packages in dist to npm package names
+const mapPackages = (files) => {
+  return files.reduce((obj, file) => {
+    const mapped = /openfn-(.+)-\d+\.\d+\.\d+\.tgz/.exec(file)
+    if (mapped && mapped[1]) {
+      obj[`@openfn/${mapped[1]}`] = path.resolve('dist', file);
+    }
+    return obj;
+  }, {});
+}
+
+findPackages().then(async (files) => {
+  const pkgs = mapPackages(files);
+  Promise.all(files.map((f) => updatePkg(pkgs, f))).then(() => {
+    const cliPath = getLocalTarballName(pkgs['@openfn/cli']);
+    console.log();
+    console.log('Build complete!');
+    console.log(`Install the CLI from ${cliPath} with the command below:`)
+    console.log();
+    console.log(`   npm install -g ${cliPath}`)
+  })
+})
+

examples/README.md

@@ -0,0 +1,22 @@
+## Examples
+
+The example apps serve to illustrate how these packages can be used, and also
+for development, any changes detected in the dependencies will trigger a rebuild in the example.
+
+**ProjectSpace Flow**
+
+Uses packages/workflow-diagram
+
+From root:
+```
+pnpm run -C examples/flow start
+```
+
+**DTS Inspector**
+
+Uses packages/describe-package
+
+From root:
+```
+pnpm run -C examples/describe-package start
+```