Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runner fails when default resources are provider per worker #14

Closed
ml-evs opened this issue Aug 10, 2023 · 3 comments · Fixed by #15
Closed

Runner fails when default resources are provider per worker #14

ml-evs opened this issue Aug 10, 2023 · 3 comments · Fixed by #15

Comments

@ml-evs
Copy link
Member

ml-evs commented Aug 10, 2023

I have a config that specifies some default resources for a given worker, e.g.,

workers: 
  my_worker: 
    type: remote
    host: example.org
    resources:
       queue_name: long_queue
       account: budget123
       time_limit: 3600

This config will lead to an error like ValueError: The following keys are not present in the template: time_limit, processes, queue_name, as currently the QResources field names in the config are not being mapped to the queuing system-specific field names, as the resources field in the worker config is just a dict rather than a QResources. I will try to make this change and see what happens.

@gpetretto
Copy link
Contributor

Indeed this is a known limitation. I esitated to include QResources as an option in the resources because you then need to somehow specify whether the dict represents a simple dict or should first be converted to a QResources. This could be handled with a type field like the Worker or the JobStore, but I was not sure if it was worth the additional complication. And maybe QResources should be converted to BaseModel?
Did you already try to implement it? Or do you have an alternative suggestion? Otherwise I will give it a try.

@davidwaroquiers
Copy link
Member

Also "somehow" related to #10 which was mentioned by @arosen93 . I did not find this PSI/J while looking for python software for queue management but I think this package is worth looking at for the future of jobflow-remote. What do you think ?

@ml-evs
Copy link
Member Author

ml-evs commented Aug 16, 2023

I implemented it in #15 (as discussed in the PR) but I was mistaken about how it was currently working. It would be nice to specify QResources in the config directly so that queue-agnostic defaults can be provided, but it works for now by just specifying the e.g., slurm-specific vars which is enough for me. I guess I'll close this as wontfix!

@ml-evs ml-evs closed this as not planned Won't fix, can't repro, duplicate, stale Aug 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants