BZ#1184630 - deployments run forever

[jistr]

https://bugzilla.redhat.com/show_bug.cgi?id=1184630

Implement timeouts for actions which do polling. Note that if DynFlow
executor stops working, the timeout checking will stop working too,
and the deployment will still run forever. This patch mitigates issues
like Puppet not being triggered, provisioning failures etc., but not
DynFlow failures.

[jistr]

Might need discussion with mburns re which BZ we'd associate it with and if we want to merge it now. (The linked BZ's objective was to mitigate the IOError issue, however this issue cannot be mitigated from Staypuft. The timeouts still would catch quite a bit of other "deployment runs forever" problems -- e.g. failed provisioning or deadlocked puppet runs.)

[jistr]

Ok, ready for review :)

[mburns72h]

I've update the bz information and added a comment that we're including this anyway.

ACK on timeouts as set. Functionality looks good visually.

[sseago]

I'm not totally clear on the under-the-covers dynflow stuff that this interacts with, but it looks reasonable on the surface here. visual ACK.

[mburns72h]

One thing i might like to see is a helpful message saying something like:

You've reached the timeout set for this action. If the action is still ongoing, you can click on the "Resume Deployment" button to continue.

[knowncitizen]

Would love to see a message, but ACK otherwise.

[jistr]

Updated with the message. Didn't redeploy to test it, but tested by copy pasting the changed part to irb. On the first try i forgot one + sign but all should be fine now :)

2.0.0-p598 :001 > module Staypuft
2.0.0-p598 :002?>   class Exception
2.0.0-p598 :003?>     end
2.0.0-p598 :004?>   end
 => nil 
2.0.0-p598 :005 > def fail(exc, text)
2.0.0-p598 :006?>   puts text
2.0.0-p598 :007?>   end
 => nil 
2.0.0-p598 :008 >             fail(::Staypuft::Exception,
2.0.0-p598 :009 >                      "You've reached the timeout set for this action. If the " +
2.0.0-p598 :010 >                      "action is still ongoing, you can click on the "
2.0.0-p598 :011?>                    "\"Resume Deployment\" button to continue.")
SyntaxError: (irb):11: syntax error, unexpected tSTRING_BEG, expecting ')'
                 "\"Resume Deployment\" button to continue.")
                  ^
(irb):11: syntax error, unexpected ')', expecting end-of-input
    from /home/jistr/.rvm/rubies/ruby-2.0.0-p598/bin/irb:12:in `<main>'
2.0.0-p598 :012 >             fail(::Staypuft::Exception,
2.0.0-p598 :013 >                      "You've reached the timeout set for this action. If the " +
2.0.0-p598 :014 >                      "action is still ongoing, you can click on the " +
2.0.0-p598 :015 >                      "\"Resume Deployment\" button to continue.")
You've reached the timeout set for this action. If the action is still ongoing, you can click on the "Resume Deployment" button to continue.
 => nil 
2.0.0-p598 :016 > 

[knowncitizen]

NICE! 2nd ACK

[mburns72h]

ack