Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime Engine #404

Merged
merged 94 commits into from
Oct 31, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
5be95d4
engine: drop in some typings
josephjclark Oct 6, 2023
07446be
engine: refactor the runners to not be factories
josephjclark Oct 6, 2023
be11be6
engine: part way through huge refactor
josephjclark Oct 10, 2023
30b70a0
Implement and test execute
josephjclark Oct 10, 2023
95a8511
engine: sort out api and engine, add unit tests on internal engine
josephjclark Oct 11, 2023
d7c7188
engine: use a context object
josephjclark Oct 11, 2023
78cc25e
engine: extra test
josephjclark Oct 11, 2023
b18c1c0
engine: tidying and typings
josephjclark Oct 11, 2023
7cda1fd
engine: add public api wrapper
josephjclark Oct 12, 2023
8e4a685
engine: fix types and restore types test
josephjclark Oct 12, 2023
90eca70
engine: tidyups
josephjclark Oct 12, 2023
8cd3008
engine: add pack
josephjclark Oct 12, 2023
de9d597
engine: api tests, fix ava issue
josephjclark Oct 13, 2023
f8fa02f
engine: more tests, fix tests
josephjclark Oct 13, 2023
ccd4eda
engine: split internal and external events
josephjclark Oct 13, 2023
195f098
runtime: add callbacks for job execution
josephjclark Oct 13, 2023
99435b6
runtime: introduce a singly notify callback
josephjclark Oct 13, 2023
d4fbcfa
runtime: add handlers for lazy loading credentials and state
josephjclark Oct 13, 2023
d3580cb
engine: lazy load intial state
josephjclark Oct 17, 2023
65b87e5
engine: typings
josephjclark Oct 17, 2023
a4fe705
mock-lightning: break out the lightning mock into its own server. We …
josephjclark Oct 17, 2023
cd0f3f3
lightning-mock: move over socket server tests
josephjclark Oct 17, 2023
397b0a9
ws-worker: move lightning mock into a different package
josephjclark Oct 17, 2023
c138b4b
worker: fix close()
josephjclark Oct 17, 2023
b5ad0c0
lightning-mock: new events
josephjclark Oct 17, 2023
ef5883d
engine-multi: ensure listen can be called before execute
josephjclark Oct 17, 2023
697533a
worker-tests: added some basic integration tests
josephjclark Oct 17, 2023
8d1683e
engine: tidying
josephjclark Oct 17, 2023
224f8d9
lightning-mock: remove logging
josephjclark Oct 17, 2023
84d3d20
cli-tests: update package name
josephjclark Oct 17, 2023
4db3042
engine: emit event after autoinstall
josephjclark Oct 17, 2023
fdf4785
worker: fix events for autoinstall
josephjclark Oct 17, 2023
2cb974a
engine: refactor ExecutionContext into its own file
josephjclark Oct 17, 2023
2efbb06
worker-tests: add autoinstall test
josephjclark Oct 17, 2023
d736b46
runtime: update job start and complete events
josephjclark Oct 17, 2023
f5cf94f
engine: map job start and complete events properly
josephjclark Oct 17, 2023
08c44bb
lightning-mock: update tests
josephjclark Oct 18, 2023
8244663
runtime: be less eager to default initial state
josephjclark Oct 18, 2023
bc01d18
various: update worker integration tests
josephjclark Oct 18, 2023
7136bc8
integration-tests: dockerise worker tests, update command to run mult…
josephjclark Oct 18, 2023
fec2405
various: fix initial state in worker, upate typings
josephjclark Oct 18, 2023
f375765
update lockfile
josephjclark Oct 18, 2023
3b7317b
bump node versions
josephjclark Oct 18, 2023
c688468
lightning-mock: quick type fix
josephjclark Oct 18, 2023
13554c3
engine: don't take an id
josephjclark Oct 18, 2023
ff5b7e9
worker: better error handling
josephjclark Oct 18, 2023
6705d71
worker: map error event to complete
josephjclark Oct 18, 2023
ee0830a
worker-tests: handle an error
josephjclark Oct 18, 2023
808d6b6
trypings
josephjclark Oct 18, 2023
9b16a8d
engine: preload credentials instead of loading on demand
josephjclark Oct 18, 2023
466ff01
types
josephjclark Oct 18, 2023
17ebba6
worker-tests: add a sort of working credential test
josephjclark Oct 18, 2023
13e824d
types
josephjclark Oct 18, 2023
8e9251d
runtime: support disabled edges
josephjclark Oct 19, 2023
6075065
worker: handle incoming enabled flag
josephjclark Oct 19, 2023
ed34d04
engine: resolveCredential should be singular
josephjclark Oct 19, 2023
3f62a4b
various: fixing resolver types and values
josephjclark Oct 19, 2023
525745a
worker: fix more typings
josephjclark Oct 19, 2023
3f21314
engine: lock down child process.env
josephjclark Oct 19, 2023
e4a4f86
workers: skip flaky test
josephjclark Oct 19, 2023
f8517b8
worker: add API to close workers
josephjclark Oct 19, 2023
4fb9388
engine: make engine creation async
josephjclark Oct 19, 2023
8e0826f
ws-worker: handle async engine creation
josephjclark Oct 19, 2023
fe17e9a
worker-tests: adapt to async engine
josephjclark Oct 19, 2023
6348a34
engine: run a little validation script on startup
josephjclark Oct 19, 2023
5f708e3
worker: types
josephjclark Oct 19, 2023
8937fec
engine: add workerpool tests, update docs
josephjclark Oct 20, 2023
266128e
engine: more security tests
josephjclark Oct 20, 2023
269ec33
engine: update worker functions
josephjclark Oct 20, 2023
0e2d510
engine: allow min-max workers to be passed
josephjclark Oct 20, 2023
2f07d90
engine: pass whitelist down from the API and expand run options to in…
josephjclark Oct 20, 2023
488ebfa
engine: do not try to install blacklisted modules
josephjclark Oct 20, 2023
673cc11
engine: update repo env var; big docs update
josephjclark Oct 20, 2023
f95d589
worker docs
josephjclark Oct 20, 2023
7e4529e
logger: export sanitize policies
josephjclark Oct 20, 2023
643e35a
worker: feed attempt options through to engine.execute
josephjclark Oct 20, 2023
d5ad41b
engine: support sanitize option (and a bit of option reworking)
josephjclark Oct 20, 2023
c7d3321
worker-tests: experimental dummy repo
josephjclark Oct 24, 2023
8e1d9e0
worker-tests: get test working (and failing sadly)
josephjclark Oct 24, 2023
881bf90
worker: tests to prove that worker threads isolate modules
josephjclark Oct 27, 2023
3e378c0
engine: experiment with terminating threads. Didn't work.
josephjclark Oct 27, 2023
4631699
runtime: support a cacheKey in linker opts
josephjclark Oct 27, 2023
d2360d4
runtime: changeset
josephjclark Oct 27, 2023
4686508
worker: use plan id as a cache key
josephjclark Oct 27, 2023
107ed75
worker-tests: add stateful adaptor test
josephjclark Oct 27, 2023
1117ec8
engine: support timeout on the attempt
josephjclark Oct 31, 2023
a6d5318
worker-tests: restore tests
josephjclark Oct 31, 2023
e24df80
runtime: comment
josephjclark Oct 31, 2023
f10fdf7
engine: remove test event
josephjclark Oct 31, 2023
caab7a7
engine: try to purge workers after every run
josephjclark Oct 31, 2023
bc8f531
docs
josephjclark Oct 31, 2023
e4a16e6
engine: prep for release
josephjclark Oct 31, 2023
719ff4b
pnpm: exclude dummy repo from workspace
josephjclark Oct 31, 2023
dc4a197
Engine: refactor timeout error
josephjclark Oct 31, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/cyan-worms-clap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@openfn/logger': patch
---

Export SanitizePolicies type
5 changes: 5 additions & 0 deletions .changeset/short-pens-punch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@openfn/runtime': patch
---

Support a cacheKey to bust cached modules in long-running processes
5 changes: 5 additions & 0 deletions .changeset/tough-coats-unite.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@openfn/runtime': patch
---

Trigger callbacks on job start, complete and init
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
# Specify the execution environment. You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub.
# See: https://circleci.com/docs/2.0/configuration-reference/#docker-machine-macos-windows-executor
docker:
- image: cimg/node:18.12
- image: cimg/node:18.18
resource_class: medium
# Add steps to the job
# See: https://circleci.com/docs/2.0/configuration-reference/#steps
Expand Down
2 changes: 1 addition & 1 deletion .tool-versions
Original file line number Diff line number Diff line change
@@ -1 +1 @@
nodejs 18.12.1
nodejs 18.18.2
4 changes: 2 additions & 2 deletions integration-tests/cli/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@openfn/integration-tests",
"name": "@openfn/integration-tests-cli",
"private": true,
"version": "0.0.1",
"version": "1.0.0",
"description": "CLI integration tests",
"author": "Open Function Group <admin@openfn.org>",
"license": "ISC",
Expand Down
2 changes: 2 additions & 0 deletions integration-tests/worker/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
node_modules
repo
15 changes: 15 additions & 0 deletions integration-tests/worker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# This dockerfile allows us to run tests against the build in isolation
# (I'm not really sure we need it but it's been a useful dev tool)
FROM node:18.12.0

ENV NODE_ENV=production

WORKDIR /app

COPY . .

COPY ./package.json.container ./package.json

RUN npm install --production

CMD ["npm", "test"]
13 changes: 13 additions & 0 deletions integration-tests/worker/ava.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
export default {
extensions: {
ts: 'module',
},

environmentVariables: {
TS_NODE_TRANSPILE_ONLY: 'true',
},

nodeArguments: ['--loader=ts-node/esm', '--no-warnings'],

files: ['test/**/*test.ts'],
};

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions integration-tests/worker/dummy-repo/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"name": "test-repo",
"private": true,
"version": "1.0.0",
"dependencies": {
"@openfn/stateful-test_1.0.0": "@npm:@openfn/stateful-test@1.0.0"
}
}
35 changes: 35 additions & 0 deletions integration-tests/worker/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"name": "@openfn/integration-tests-worker",
"private": true,
"version": "1.0.0",
"description": "Lightning WOrker integration tests",
"author": "Open Function Group <admin@openfn.org>",
"license": "ISC",
"type": "module",
"scripts": {
"clean": "rimraf dist tmp/repo/*",
"build:pack": "pnpm clean && cd ../.. && pnpm pack:local integration-tests/worker/dist --no-version",
"build": "pnpm build:pack && docker build --tag worker-integration-tests .",
"start": "docker run worker-integration-tests",
"test": "pnpm clean && npx ava -s --timeout 2m"
},
"dependencies": {
"@openfn/engine-multi": "workspace:^",
"@openfn/lightning-mock": "workspace:^",
"@openfn/logger": "workspace:^",
"@openfn/ws-worker": "workspace:^",
"@types/node": "^18.15.13",
"@types/rimraf": "^3.0.2",
"ava": "5.3.1",
"date-fns": "^2.30.0",
"koa": "^2.13.4",
"rimraf": "^3.0.2",
"ts-node": "10.8.1",
"tslib": "^2.4.0",
"typescript": "^5.1.6"
},
"files": [
"dist",
"README.md"
]
}
34 changes: 34 additions & 0 deletions integration-tests/worker/package.json.container
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"name": "@openfn/integration-tests-worker",
"private": true,
"version": "1.0.0",
"description": "Lightning WOrker integration tests",
"author": "Open Function Group <admin@openfn.org>",
"license": "ISC",
"type": "module",
"scripts": {
"clean": "rimraf dist tmp/repo/*",
"build:pack": "npm run clean && cd ../.. && npm run pack:local integration-tests/worker/dist --no-version",
"build": "npm run build:pack && docker build --tag worker-integration-tests .",
"start": "docker run worker-integration-tests",
"test": "npm run clean && npx ava -s --timeout 2m"
},
"dependencies": {
"@openfn/engine-multi": "./dist/openfn-engine-multi.tgz",
"@openfn/lightning-mock": "./dist/openfn-lightning-mock.tgz",
"@openfn/logger": "./dist/openfn-logger.tgz",
"@openfn/ws-worker": "./dist/openfn-ws-worker.tgz",
"@types/node": "^18.15.13",
"ava": "5.3.1",
"date-fns": "^2.30.0",
"rimraf": "^3.0.2",
"ts-node": "10.8.1",
"tslib": "^2.4.0",
"typescript": "^5.1.6",
"@types/rimraf": "^3.0.2"
},
"files": [
"dist",
"README.md"
]
}
1 change: 1 addition & 0 deletions integration-tests/worker/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Integration tests for Lightning Worker architecture.
Loading