Fixes #29815 - Fixed the runner to execute rescue scenario

[patilsuraj767]

Issue -
Rescue scenario never executed if the backup fails in the middle due to any reason. Not only for backup but in general rescue scenario is never triggered. (The main scenario fails with error - "The runner is already in quit state".)

Root cause -
When the main scenario fails due to any reason, it triggers the rescue scenario. But in 'execute_scenario_steps' method 'run_scenario' is called for running the before_scenarios(if any). Then in run_scenario expectation is raised 'The runner is already in quit state' if quit? (quit? is set from main scenario as it failed), this quit? is stoping rescue scenario from running.

Reproducer steps -

1. Make sure backup will fail by editing /etc/foreman-proxy/settings.d/dhcp.yml. modify it as below.

:use_provider: dhcp_infoblox

Instead of dhcp_isc make it dhcp_infoblox

2. run backup

# mkdir /tmp/backup/
# foreman-maintain backup offline --assumeyes --skip-pulp-content /tmp/backup/

Backup should fail with

Backup config files:                                                  [FAIL]
Couldn't find DHCP Configuration file

Note - Above reproducer steps will only work till the merge of this PR

[theforeman-bot]

Issues: #29815

[upadhyeammit]

Hello Suraj,

At the time of backup I can see rescue scenario getting executed if I stop mongod service in between, so I think this issue is not generic and its specific to some use cases ?

| Saving metadata to metadata.yml                                     [OK]      
--------------------------------------------------------------------------------
Backup config files: 

| Collecting config files to backup                                   [OK]      
--------------------------------------------------------------------------------
Backup Pulp data:                                                     [SKIPPED]
--------------------------------------------------------------------------------
Backup Mongo online: 
| Getting dump of Mongo DB                                            [FAIL]    
Failed executing scl enable rh-mongodb34 -- mongodump --host localhost --port 27017 -d pulp_database --out /tmp/satellite-backup-2020-06-17-19-07-21/mongo_dump, exit status 1:
 2020-06-17T19:07:37.697+0530	Failed: error connecting to db server: no reachable servers
--------------------------------------------------------------------------------
Scenario [Backup] failed.

The following steps ended up in failing state:

  [backup-online-mongo]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="backup-online-mongo"



Running Failed backup cleanup
================================================================================
Clean up backup directory:                                            [OK]
--------------------------------------------------------------------------------

Done with backup: 2020-06-17 19:07:37 +0530
Backup didn't finish. Incomplete backup was removed.

[upadhyeammit]

@patilsuraj767 Hello, any further updates on this ?