New runtime manager service

[josephjclark]

The Runtime Manager Service is a long-lived, multi-threaded out-of-process worker to support Lightning. It will initially be hosted inside a runtime manager server, which provides the HTTP interface between lightning and the runtime manager.

The service itself is responsible for running jobs and workflows in worker threads (or whatever!), re-directing console.log messages to an event bus, and caching (probably just in memory) compiled job expressions. It also needs to provide a reporting API to communicate memory usage, thread counts, status of jobs, etc.

It will need to auto-install adaptors and cache compiled jobs.

The service should ideally have zero persistence.

Worker threads should not allow any file i/o within a job, and we need good unit tests to enforce and prove this.

We've started to build this out in packages/runtime-manager.

This issue builds on the new runtime (#11) , but does not include the webserver that will talk to Lightning. That will be covered in another issue (#201)

[josephjclark]

A really important job here now is to support the runtime repo. This means the runtime manager should support and auto-install into a repo, and pass the path through to the runtime.

At the time of writing, any job imports are basically gonna break as a result of #147

When this is done, we need to revisit the slow-random test job and restore the common import.

[taylordowns2000]

@josephjclark , is this the right issue to link with #193 ? (And should we make it an epic?)

[josephjclark]

@taylordowns2000 I know you said to forget the admin but I'm still in high-level design mode. This issue has been updated to reflect the current architecture.

[josephjclark]

Closing this down as work has progressed way beyond this issue (which was always a high level design issue)