Randomly select Relay at beginning of stage only #790
Conversation
This commit modifies the Relay selection logic in Cog.Executor such that a Relay is randomly selected once at the start of each pipeline stage and is then reused for subsequent invocations for that stage. Doing so allows Cog to take maximum advantage of Relay's ability to reuse containers for the same pipeline stage and is expected to improve performance for Cog installations with multiple Relays.
|
👍 This looks reasonable to me, but I do have a question. If you are in the middle of a long running plan, and the bundle is unassigned from the relay, what happens? Would Cog just crash? |
|
The container wouldn't get torn down when the bundle is removed. Container destruction happens once it's idle. In the scenario you describe I'd expect the plan for the pipeline stage to finish without error (assuming the command invocations didn't throw any) but future invocations wouldn't select the same Relay(s). UPDATE: What I said isn't entirely accurate. The Docker environment is unreachable once the bundle has been removed from the Relay. A more accurate description of behavior is that the Relay would continue to service invocations until the bundle has been removed. When that happens it will return a "bundle not found" error back to Cog. So, yes, the pipeline will crash if that happens before the invocations have finished. |
|
This isn't going to do what we want; an invocation can generate multiple plans, each of which is completely isolated from the others in its cohort. We're going to need to remember the relay that's chosen at a higher level, probably in the executor state itself. I had a proposal for this in #777. |
|
@kevsmith cool that makes sense. I asked more out of curiosity than anything, but is that something we should actually worry about? Should we check to see if relay is still valid on each execution and pick a new relay if it isn't? |
Previous iteration assigned Relays when each plan was executed. This was wrong as a single invocation can generate multiple plans which negated the benefits of selecting a Relay when each plan executes. This commit moves Relay selection into the planner. A Relay is selected before plans for an invocation are created and shared across all the created plans.
This commit modifies the Relay selection logic in Cog.Executor such that a Relay is randomly selected once at the start of each pipeline stage and is then reused for subsequent invocations for that stage. Doing so allows Cog to take maximum advantage of Relay's ability to reuse containers for the same pipeline stage and is expected to improve performance for Cog installations with multiple Relays.