Flagging Operations as "submit only"

[csadorf]

Original report by RoseCers (Bitbucket: RoseCers, ).

---

I'd like a command line functionality to be added such that I can say "python project.py run -o all_not_for_submit --progress --parallel" so that I can run all quick aggregators without trying to run operations that should be "submit only"

[csadorf]

Original comment by RoseCers (Bitbucket: RoseCers, ).

---

Another option is providing an "--except" flag for python project.py run, where --except denotes which operations not to run @csadorf

[csadorf]

Original comment by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

---

Duplicate of #30.

[csadorf]

This is still relevant and I think we should address this as a separate issue.

[csadorf]

@rosecers I've reopened this issue that you had created a while back.

[csadorf]

I think we should have conditions that allow you to specify whether a specific operation is only eligible in certain execution contexts, such as during submission or only local runs. Here is what the API could look like:

@Project.no_submit
def only_run_or_script(job):
    pass

@Project.no_run
def only_submit_or_script(job):
    pass

@Project.no_run
@Project.no_submit
def script_only(job):
    pass

These decorators can optionally be made conditional by providing a condition function which may or may not be a function of job and an optional reason:

@Project.no_run(on_laptop, reason='do not execute locally')
@Project.no_submit(condition=lambda job: job.sp.N < 10000, reason='system size too small')
def only_submit_sometimes(job):
    pass

I am not sure how we should signal this kind of specificity in the status overview. @rosecers Thoughts?

[rosecers]

Hey Simon, that would probably work. How would it jive with the old API?

            self.add_operation(
                            name=
                            cmd=
                            pre=[],\
                            post=[])

I really like this API and have not switched to the new one.

[csadorf]

Hey Simon, that would probably work. How would it jive with the old API?

            self.add_operation(
                            name=
                            cmd=
                            pre=[],\
                            post=[])

I really like this API and have not switched to the new one.

I intend to keep that "old API" supported indefinitely, so we could probably add arguments no_run, no_submit, and no_script to the FlowProject.add_operation() function. Alternatively we could group them into some kind of execution directive.

Do you have any thoughts about how to display this as part of the status output?

[rosecers]

We could either separate them into columns or place flags [NR, NS, NSC] along with [I,Q,S,H,A], etc.

…

On Feb 26, 2019, at 11:17 AM, Carl Simon Adorf ***@***.***> wrote: Hey Simon, that would probably work. How would it jive with the old API? self.add_operation( name= cmd= pre=[],\ post=[]) I really like this API and have not switched to the new one. I intend to keep that "old API" supported indefinitely, so we could probably add arguments no_run, no_submit, and no_script to the FlowProject.add_operation() function. Alternatively we could group them into some kind of execution directive. Do you have any thoughts about how to display this as part of the status output? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AtVX7iy83r9A-3GAWPKpJCLkHjbkNxljks5vRV4PgaJpZM4bN93X>.

[csadorf]

We could either separate them into columns or place flags [NR, NS, NSC] along with [I,Q,S,H,A], etc.

I think adding more columns might be problematic space-wise, but I like the idea of just adding flags.

[vyasr]

Solving this will be much easier with #114. Once that is merged we can revisit whether or not to implement further conveniences to enable this feature.

[yuanzhou0827]

Thank you, Vyas!

…

On Jul 5, 2019, at 11:50 AM, vyasr ***@***.***> wrote: Solving this will be much easier with #114 <#114>. Once that is merged we can revisit whether or not to implement further conveniences to enable this feature. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#33?email_source=notifications&email_token=ALGM7ZFI76NM5BKV3NYQLOLP55UTXA5CNFSM4GZX3XL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZJ23SQ#issuecomment-508800458>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALGM7ZGA7WKF3BDIF5CQV7DP55UTXANCNFSM4GZX3XLQ>.

[vyasr]

@csadorf now that groups are merged, how do you propose we address this? Perhaps using a directive makes sense? Otherwise, we could do a separate decorator.

[bdice]

It seems to me like using the new groups feature would solve this problem. You would create a group of all the quick-running operations and run that group locally, then submit all others. I don't think there is enough reason to create an entirely new feature for this if groups will suffice.

[vyasr]

I'm perfectly fine with that solution, it's how I would probably address it myself in the current flow and I agree that there's no need to introduce additional complexity.