Fixes #27826 - Update Mongo upgrade recovery logic

[chris1984]

This PR fixes the following issues:

/usr/share/ruby/fileutils.rb:245:in `mkdir': File exists - /var/tmp/mongodb_engine_upgrade (Errno::EEXIST)
        from /usr/share/ruby/fileutils.rb:245:in `fu_mkdir'
        from /usr/share/ruby/fileutils.rb:174:in `block in mkdir'
        from /usr/share/ruby/fileutils.rb:173:in `each'
        from /usr/share/ruby/fileutils.rb:173:in `mkdir'
        from /usr/share/katello-installer-base/hooks/pre/31-mongo_storage_engine.rb:19:in `migration'
        from /usr/share/katello-installer-base/hooks/pre/31-mongo_storage_engine.rb:65:in `block (4 levels) in load'

This commit solves the disk space issue reported in the bz/case that created this:

• a667c90

• Removed moving the contents to the backup dir since we did a dump, this was causing large mongo databases to quickly fill up /var/tmp

• Added better recovery steps incase of failure

Steps to test

• Install 6.3 and upgrade to 6.4 - 6.5

• Add some content/repos

• Test the mongo upgrade

Normal run with passing results:

[root@toledomongo ~]# foreman-installer -v --upgrade-mongo-storage-engine

mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database finished successfully!
[ INFO 2019-09-11T03:21:25 verbose] All hooks in group pre finished

Normal run with folder present from a failed upgrade attempt:

mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database finished successfully!

Failure test run on trying to restore to test logic of recovery steps, at the end backup is present, mongo service is running and storage engine is in mmapv1:

[ERROR 2019-09-11T04:08:42 verbose] mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error!
[ERROR 2019-09-11T04:08:52 verbose] The restore could not be completed correctly, reverting actions.
[ INFO 2019-09-11T04:09:01 verbose] Starting MongoDB with old config and database files
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod
- All services started                                                [OK]      
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!
[ERROR 2019-09-11T04:09:05 verbose] Mongo started up in MMAPV1 mode, backup at /var/tmp/mongodb_engine_upgrade
[ INFO 2019-09-11T04:09:05 verbose] Installer finished in 72.779767646 seconds

[root@toledomongo ~]# ll /var/tmp/mongodb_engine_upgrade/
total 4
drwxr-xr-x. 2 root root 4096 Sep 11 04:08 pulp_database

[root@toledomongo ~]# service rh-mongodb34-mongod status
Redirecting to /bin/systemctl status rh-mongodb34-mongod.service
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2019-09-11 04:09:05 UTC; 1min 4s ago

> db.serverStatus().storageEngine
{
	"name" : "mmapv1",
	"supportsCommittedReads" : false,
	"readOnly" : false,
	"persistent" : true
}

[chris1984]

@ehelms can you review this, this is one of the bugs for 6.6