-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve runner API #296
Improve runner API #296
Conversation
The struct methods have been rearranged, to avoid increasing too much the complexity, in particular for no special purpose
I'm completing the tests and the review of the runner API. The rest should be already alright (provided that tests are still passing). To make @felixhekhorn happier: I dropped |
Since I'm reviewing the runner API: may I drop the |
I completed the review of the But before merging I'd like to have a plan for the legacy runner removal. This whole PR was meant to provide tools for Pineko to implement a further runner, similar to the managed one, but more sophisticated. There should be no reason to keep the legacy any further. EDIT: now discussed #398 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine. However, since this is surely part of the v0.14 series we should start merging the related PRs (order to be determined), since we have now several PRs touching these files (e.g. #291 )
I was already on it |
The idea is to allow the definition of a suitable runner in Pineko, to make use of the
Q2
granularity, and eventually dropn_integration_cores
in EKO.If we can suitably split at the level of
Q2
, there is no need to parallelize at the level ofx
, since ourxgrid
is not going to grow wild. Avoiding parallelization will simplify a bit the content of theeko
package, and it will solve many problems encountered with Python parallelization.To allow this, the following steps are required:
eko.runner
, such that it will be reliable outside the package itself (this will be the public runner API) -> postponed to Expose new runner API #399Q2
values, for which we only needNone
, the default, use a temporary one, as it is currently done)In a later PR, a merge utility can be added nevertheless, taking into account the scenario in which a single EKO is initialized, and copied over for parts and operators computation.
In particular, this limited
merge
utility will take two EKO tar files, and just copy the content ofparts/
andoperators/
subfolders of the second into the first one (in-place operation, to save disk - to implement the immutable operation just copy the first EKO tar beforehand).