Runtime Engine

[josephjclark]

This branch is tracking changes needed to update and integrate the multi-threaded runtime engine

• Restructure API (this was a big deal)
• Support job start and complete in the runtime
• Pull the lightning mock out into its own package
• Integration tests of lightning mock, worker and engine-multi running locally
• Containerised integration tests
• Autoinstall improvements and tests
• Handling lazy-loading initial state
• Handling lazy-loading credentials
• Whitelist @openfn/modules
• Better security on pooled threads runtime-manager: lock down pooled threads #120
• Support disabled edges RTM: support disabled edges #263
• Support options on attempt (including sanitize arguments)
• Kill timeout threads RTM: Kill a thread that timesout (or maybe stops responding to a heartbeat) #365
• validate worker thread with a handshake event, throw if it it fails Epic: Runtime Engine #393 (comment)
• Check that the parent env is hidden to the running worker Epic: Runtime Engine #393 (comment)
• Integration test on stateful adaptors
• Release bundles

The audit trail on this isn't going to be terribly good because I'm just keeping my head down and making it all work.

An interesting thing about the new engine design is that there's a core engine component, which exposes all of its internals in a large ish API (well, sort of), versus the API component, which wraps the engine, sets defaults and obfuscates parts of the API that no-one should be tampering with.

The idea is that clients/wrappers of the engine will instantiate the API and use its limited surface basically to a) run workflows (we call them workflows or plans here) and b) listen to events emitted during execution.

For unit tests, or for custom deployments or advanced users, you can use the engine directly. This lets us do things like use a mock worker environment, override autoinstall and compile rules.

This PR should close #393 and everything in it. Apart from the logs thing, which is going to be ongoing.

Reasons

For now, I am returning to attempt:complete with an "OK" reason

Any critical/crash errors along the way will return to attempt:complete with a "FAIL" reason.

This needs a lot more work, it's just where I am right now.

[josephjclark]

Ach a late point of confusion has crept up, I think I need to deal with this next week because my head is spinning.

Should the engine accept resolvers in its constructor (one set of resolvers works for all credentials and dataclips), or should they be passed to each execute call?

At the moment they are passed on execute, but suddenly that seems strange to me (actually, I think I've accidentally been supporting both APIs all week and the point is I need to consolidate on just one)