[BUG] feature example leaves messages in queue

[lucpeterson]

🐛 Bug Report

Describe the bug
The feature example, which shows how to stop workers from within a worker leaves the stop worker command in the queue, since the worker kills itself and doesn't acknowledge that it finished. This can cause other workers later to pick this back up and kill themselves and others later on.

To Reproduce
Steps to reproduce the behavior:

1. merlin run feature_demo.yaml
2. merlin run-workers feature_demo.yaml
3. merlin status feature_demo.yaml

--> you'll see there's still a task in the queue

Expected behavior
The queue should be empty after the workflow finishes.

Additional context
I think the fix is to add a delay to the stop workers command in the step and background it, something like

( sleep 30; merlin stop-workers ) &

instead of just merlin stop-workers

this should fork a child background process that will execute after the parent worker finishes the step (and removes it from the server). I'm not sure what the sleep delay should be.

[ben-bay]

Uh oh that's bad. Good catch

[ben-bay]

One possibly mitigating factor is that the current code reads:

merlin stop-workers --workers '.*$(WORKER_NAME)*'

...so it's limited to one particular worker name.

[ben-bay]

@lucpeterson I just tested your suggested solution, but it didn't work. The demo_worker remains

[ben-bay]

This step has been removed for the time being.

[lucpeterson]

#182 should address this