Simple resubmit with '--after' flag

[csadorf]

Original report by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

---

It should be possible to easily resubmit a job-operation that runs after the exact same operation has finished.

Something like this:

#!bash
$ python -m my_project.submit --resubmit

[csadorf]

Original comment by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

---

@vyasr Does that capture what we discussed the other day?

[csadorf]

Original comment by Vyas Ramasubramani (Bitbucket: vramasub, GitHub: vyasr).

---

Yes, that's right. I can't imagine it would be too difficult to incorporate on top of what already exists within flow.

[vyasr]

@csadorf I no longer recall what this is intended to accomplish. The --after flag exists for submission, I'm not sure what else we wanted. Do you have a better recollection, or should we close this issue?

[csadorf]

@vyasr This is about the possibility to exploit the schedulers own ability to chain multiple submissions together. I believe that this is still something that should be addressed, possibly by exploiting a 3rd party submission system. I don't think we should close this issue.

[csadorf]

This is a non-trivial issue, so I removed the good first issue label.

[vyasr]

My confusion is that the title doesn't really seem to match the description. Was the idea for the --resubmit argument that it would traverse the operation graph, figure out what should run next if everything currently submitted ran to completion, and then submit those operations using the scheduler's ability to chain submissions? flow already allows users to manually indicate that the submission of a particular job operation should include the --after OTHER_JOB flag to mark that cluster job for running only after another job is completed. Additionally, how would this relate to #114?

[csadorf]

I'm honestly not quite sure, maybe this issue is just not well defined enough and we should close it.

[bdice]

I think this may have been better-defined under an earlier execution model. When job operations map 1-1 with cluster jobs, this makes a lot of sense. The DAG of conditions often determines what operations should become eligible after the currently running operations complete, and so it's reasonable to pre-submit operations to the cluster's queue and use the scheduler to your advantage (which minimizes wait times). However, it would be quite hard to implement this and I expect it to become even more complicated with groups/aggregations. I agree that it's appropriate to close this, since no user has demanded this feature and it would be a complex implementation with no immediate use case.