Epic: Runtime Engine

[NickOpenFn]

Epic to track the work needed in the multi-process runtime worker.

Remember we don't have a good name for this (runtime manager is insufficient). See #374. Runtime Worker might be useful.

Essential

• Review the existing code and work out what needs doing
• Runtime Manager: whitelist to only allow imports of @openfn/ modules #372
• Runtime Manager: implement autoinstall #373
• runtime-manager: lock down pooled threads #120 (security, need to investigate this)
• Work out what this comment means: New runtime manager service #52 (comment)

Useful

• What do out of process logs look like? #388
• RTM: Kill a thread that timesout (or maybe stops responding to a heartbeat) #365

Optional

• RTM: Add a keepalive/heartbeat event #265 (idk if we need this any more)
• RTM: support disabled edges #263

To close

• New runtime manager service #52 is irrelevant now, I just need to look into that mysterious comment and either transfer notes to a new issue or close it down

Not in issues

Stuff that needs doing that we don't have issues for

• Support the sanitize log flag
• Does the existing service worker even support workflows? Does it need to do anything for that?
• Lazy loading assets (state/credentials)~~ (basically done but I'm testing)
• At the moment we use worker threads to run the jobs (with compilation in the main thread). I don't know if this is its final form. But we should be able to swap this out later.
• Caching compiled jobs
• Update API to meet new standards
• Log version numbers (there is a wider story about logging but right now we'll just do the minimum)

Security?

Security considerations for the actual process worker are more to do with not allowing malicious code to run and not allowing other environments/processes/jobs to be read. A lot of the security is subsumed by the enclosing server.

[josephjclark]

TODO validate incoming workflows/attempts to make sure they're what we expect.

This is good practice generally but more interestingly it'll keep the worker honest. On that note, the worker should do some validation too on the incoming attempts

[josephjclark]

Closing this down as it's no longer useful - the work has been done or branched off