-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cylc get-config: memory usage and reduce use from Rose #22
Comments
Looks like this is still an issue for large suites, as it can take |
See cylc/cylc-flow#170. |
Matt, I've just pushed my config parsing changes to the cylc repository, for cylc/cylc-flow#170, having managed to get get-config() down to under 10s for your test suite: 1000 tasks, each of which configures almost 300 environment variables, going through 4 levels of inheritance (then we have to add default values for the complete [runtime] structure, for each task). Cylc holds all task configuration data, the result of inheritance processing, in memory ready for use (and has to add in defaults for each task too, for the entire [runtime] structure). So I think it's clear, particularly with Jinja2, it will always be easy to generate a suite that will take a lot of time and/or memory to parse. If your only problematic use of get-config() in extracting task host and owner in an event handler script (is it?) I would suggest again that cylc should just add task host and owner to the argument list supplied to task event handlers. Do you have a specific objection to that? Event handlers that don't need the extra information can ignore it. |
A comment on possible further efficiency gains that may be possible: cylc/cylc-flow#170 (comment) |
Support the new syntax of "cylc get-config", which can return owner and host information of a task or all tasks in a single call.
#22: support "cylc get-config -i ITEM".
cylc/cylc-flow#86, cylc/cylc-flow#140, cylc/cylc-flow#181 may improve things. |
Several technical challenges remain. In order to pull the relevant job logs of a task at a given cycle time from a remote host, we'll need the |
@matthewrmshin - we have been storing |
As discussed, we'll need the |
@matthewrmshin - will put up a pull request shortly that does this. |
cylc/cylc-flow#296 adds the required functionality for this. |
See #477. |
Calls to
cylc get-config
can kill the server. Need to investigate, and possibly further reduce our reliance on it.This problem is less urgent now that we have reduced the number of calls of
cylc get-config
to 1 per task event. If and when we have the database for Cylc task event or status, we should be able to make further improvement to this.The text was updated successfully, but these errors were encountered: