Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New runtime manager service #52

Closed
josephjclark opened this issue Oct 28, 2022 · 4 comments
Closed

New runtime manager service #52

josephjclark opened this issue Oct 28, 2022 · 4 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@josephjclark
Copy link
Collaborator

josephjclark commented Oct 28, 2022

The Runtime Manager Service is a long-lived, multi-threaded out-of-process worker to support Lightning. It will initially be hosted inside a runtime manager server, which provides the HTTP interface between lightning and the runtime manager.

The service itself is responsible for running jobs and workflows in worker threads (or whatever!), re-directing console.log messages to an event bus, and caching (probably just in memory) compiled job expressions. It also needs to provide a reporting API to communicate memory usage, thread counts, status of jobs, etc.

It will need to auto-install adaptors and cache compiled jobs.

The service should ideally have zero persistence.

Worker threads should not allow any file i/o within a job, and we need good unit tests to enforce and prove this.

We've started to build this out in packages/runtime-manager.

This issue builds on the new runtime (#11) , but does not include the webserver that will talk to Lightning. That will be covered in another issue (#201)

@josephjclark josephjclark mentioned this issue Oct 28, 2022
41 tasks
@josephjclark josephjclark added the enhancement New feature or request label Oct 28, 2022
@josephjclark
Copy link
Collaborator Author

A really important job here now is to support the runtime repo. This means the runtime manager should support and auto-install into a repo, and pass the path through to the runtime.

At the time of writing, any job imports are basically gonna break as a result of #147

When this is done, we need to revisit the slow-random test job and restore the common import.

@taylordowns2000
Copy link
Member

@josephjclark , is this the right issue to link with #193 ? (And should we make it an epic?)

@taylordowns2000 taylordowns2000 changed the title Runtime Manager: next steps New runtime manager service Mar 24, 2023
@josephjclark
Copy link
Collaborator Author

@taylordowns2000 I know you said to forget the admin but I'm still in high-level design mode. This issue has been updated to reflect the current architecture.

@josephjclark
Copy link
Collaborator Author

Closing this down as work has progressed way beyond this issue (which was always a high level design issue)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants