[WIP] Run automate calls thru queue

[mkanoor]

Automate engine will be called via Queue to be picked by
Automate workers in the correct zone.

https://bugzilla.redhat.com/show_bug.cgi?id=1231939

[mkanoor]

@Fryguy @gmcculloug @tinaafitz
Please review

This depends on the PR #3215

[miq-bot]

<stalled_finished_job />Detected and restarted stalled travis job.

[kbrock]

I just deleted a bunch of "rerunning for travis"

a) I don't think the timeout code is working in this code. Travis is obviously deadlocking.
b) removed race condition: #3266
c) added ability to skip the queue: #3268

[Fryguy]

One thing you can do is just close this PR until you have a fix for the deadlock, then reopen

[Fryguy]

cc @jrafanie Look what you did!

[jrafanie]

Ah, that could explain the Travis backlog. As long as you kill the job before it's marked as stalled, the bot shouldn't restart it.

Looks like a feature request on the bot.

[jrafanie]

Let's reopen when the tests are fixed.

[kbrock]

@mkanoor @gmcculloug In the code base, wait_for_taskid needs to be removed. It blocks a thread/worker until a task is performed. Running tasks synchronous have the same issue. If you use this, then the worker will just block, not doing any other jobs while blocked.

Again: If we are waiting on a user to click a button, then no other automate tasks will run on this worker, only a sleep.

Instead, we would like to encourage:

a) using regular jobs instead of tasks.
b) removing synchronous behavior and splitting these tasks into multiple methods.

For our needs here, I don't know why we are using either of these mechanisms. And combining both together seems to multiply the negative side effects.

Also, a single task id of "automate" means that there can only be 1 worker that performs all automate work. So blocking on user input will effectively stop all of automate from running.

[kbrock]

please dup the options coming in since you are modifying them

[mkanoor]

@kbrock @jrafanie @gmcculloug @Fryguy
Trying to see if this resolves the deadlock on Travis

[miq-bot]

Checked commits mkanoor/manageiq@d1c6f7c~...676f704 with ruby 1.9.3, rubocop 0.34.2, and haml-lint 0.13.0
8 files checked, 8 offenses detected

vmdb/lib/miq_automation_engine/engine/miq_ae_engine.rb

• 🔶 - Line 32, Col 12 - Style/ExtraSpacing - Unnecessary spacing detected.
• 🔶 - Line 355, Col 9 - Style/ExtraSpacing - Unnecessary spacing detected.
• 🔹 - Line 30, Col 3 - Metrics/AbcSize - Assignment Branch Condition size for deliver_synchronous is too high. [26.27/15]
• 🔹 - Line 45, Col 121 - Metrics/LineLength - Line is too long. [129/120]
• 🔹 - Line 305, Col 3 - Metrics/AbcSize - Assignment Branch Condition size for create_automation_args is too high. [44.34/15]
• 🔹 - Line 305, Col 3 - Metrics/CyclomaticComplexity - Cyclomatic complexity for create_automation_args is too high. [7/6]
• 🔹 - Line 305, Col 3 - Metrics/MethodLength - Method has too many lines. [26/25]
• 🔹 - Line 305, Col 3 - Metrics/PerceivedComplexity - Perceived complexity for create_automation_args is too high. [8/7]

[kbrock]

just deleted 15 stalled request comments.
Lets hold off rebasing this until we have a way around hanging travis. We are having enough problems with it right now.

also, We'll need to ensure that the automate worker is enabled for this to work.

[mkanoor]

Closed temporarily