Epic: Runtime Manager Backend

[josephjclark]

Epic to track the work needed in the multi-process runtime worker.

Remember we don't have a good name for this (runtime manager is insufficient). See #374. Runtime Worker might be useful.

Essential

• Review the existing code and work out what needs doing
• Runtime Manager: whitelist to only allow imports of @openfn/ modules #372
• Runtime Manager: implement autoinstall #373
• runtime-manager: lock down pooled threads #120 (security, need to investigate this)
• Work out what this means: New runtime manager service #52 (comment)

Useful

• What do out of process logs look like? #388
• RTM: Kill a thread that timesout (or maybe stops responding to a heartbeat) #365

Optional

• RTM: Add a keepalive/heartbeat event #265 (idk if we need this any more)
• RTM: support disabled edges #263

To close

• New runtime manager service #52 is irrelevant now, I just need to look into that mysterious comment and either transfer notes to a new issue or close it down

Not in issues

Stuff that needs doing that we don't have issues for

• Support the sanitize log flag
• Does the existing service worker even support workflows? Does it need to do anything for that?
• Lazy loading assets (state/credentials)
• At the moment we use worker threads to run the jobs (with compilation in the main thread). I don't know if this is its final form. But we should be able to swap this out later.
• Caching compiled jobs

Security?

Security considerations for the actual process worker are more to do with not allowing malicious code to run and not allowing other environments/processes/jobs to be read. A lot of the security is subsumed by the enclosing server.

[NickOpenFn]

Closing as a duplicate of the Epic: #364