Fix Issue #196: Filtered the workflow context sent to worker by server

[parauliya]

Description

There are following change in this PR:

1. Filtered the workflow context sent to worker by server on the basis of status and worker_id.
2. Changed Info logs to Debug during sleep in worker side
3. Changed a bit of logic in worker for not repeating the same workflow execution again.

Following things are fixed in second commit of this PR

1. Modified the filter from the server side while sending the workflow context
2. Added a gRPC stream for getting the workflow context
3. Continue the loop for worker even after completing the workflow so that it can fetch any new workflow.

Why is this needed

This resolves the following issue.
Fixes: #196 #222 #72

How Has This Been Tested?

This has been tested by creating a sample worklfow and execute it.
Following scenarios was executed for the testing:

1. One workflow has one worker
2. One workflow has two workers
3. Two workflows and each workflow has two workers

How are existing users impacted? What migration steps/scripts do we need?

It will enhance the user experience as logging will be lesser as compared to the current one on worker side.
There is no migration script needed.

[nathangoulding]

We still need to handle the scenarios where we have an already connected worker/stream, and 1) a new workflow gets created that targets that worker, or 2) an already executing workflow reaches a step where the worker gets new work. How is this achieved in this situation?

[parauliya]

We still need to handle the scenarios where we have an already connected worker/stream, and 1) a new workflow gets created that targets that worker, or 2) an already executing workflow reaches a step where the worker gets new work. How is this achieved in this situation?

This handles both of your cases as follows:

1. When the worker do the iPXE boot for the first time, it first executes all the existing workflow which are targeted to it and then it will keep on running on that worker machine. And If you create a new workflow which is targeted to this worker then that workflow will be executed automatically.
2. Worker will first execute all the workflows which were created before it was started and then it will go for the new workflows which are targated to it.

[nathangoulding]

This handles both of your cases as follows:

1. When the worker do the iPXE boot for the first time, it first executes all the existing workflow which are targeted to it and then it will keep on running on that worker machine. And If you create a new workflow which is targeted to this worker then that workflow will be executed automatically.
2. Worker will first execute all the workflows which were created before it was started and then it will go for the new workflows which are targated to it.

You're right. When we add the ability for workers to fetch-and-wait instead of fetch-and-quit, as addressed in issue #222, we will need to update this so that the server sends work to the workers that are already connected and waiting.