zaza continues with next test in suite despite model not being torn down (holds resources, etc.)

[ajkavanagh]

A model that errors is not fully torn down and may hold on to resources (e.g. a vip). This means that the following bundle in the suite will fail if it uses the same vip (which it probably will). The model needs to be force-removed.

However, equally, zaza should NOT continue if the model can't be torn down. There's no point in continuing.

example of failure:

Model              Controller      Cloud/Region             Version  SLA          Timestamp  Notes
zaza-eb5f8cc00b42  auto-osci-sv03  serverstack/serverstack  2.8.7    unsupported  14:06:46Z  attempt 2 to destroy model failed (will retry):  model not empty, found 3 machines, 3 applications (model not empty)

App              Version  Status       Scale  Charm            Store       Rev  OS      Notes
hacluster                 maintenance      3  hacluster        jujucharms  160  ubuntu  
keystone                  waiting          0  keystone         jujucharms  528  ubuntu  
percona-cluster  5.7.20   error            3  percona-cluster  local        45  ubuntu  

Unit                Workload     Agent      Machine  Public address  Ports     Message
percona-cluster/0*  error        idle       1        172.17.103.20   3306/tcp  hook failed: "cluster-relation-departed"
  hacluster/2*      maintenance  executing           172.17.103.20             (stop) stopping charm software
percona-cluster/1   blocked      executing  2        172.17.103.21   3306/tcp  MySQL is down. Sequence Number: -1. Safe To Bootstrap: 0
  hacluster/1       maintenance  executing           172.17.103.21             (stop) stopping charm software
percona-cluster/2   blocked      executing  3        172.17.103.19   3306/tcp  Insufficient peers to bootstrap cluster
  hacluster/0       maintenance  executing           172.17.103.19             (stop) stopping charm software

Machine  State    DNS            Inst id                               Series  AZ    Message
1        stopped  172.17.103.20  099a4e91-1b45-4a6f-97aa-1c8bb1eca12b  bionic  nova  ACTIVE
2        stopped  172.17.103.21  b46d599c-cfc4-4d28-bac4-7ec40897f493  bionic  nova  ACTIVE
3        stopped  172.17.103.19  3309e7ee-7f4f-4fdc-a2cc-570eea9f4331  bionic  nova  ACTIVE

And result of it in following model:

2021-01-19 14:06:40 [INFO] vip (172.17.103.230) running in percona-cluster/0
2021-01-19 14:06:40 [INFO] New crm_master unit detected on percona-cluster/0
2021-01-19 14:06:40 [ERROR] connection refused connecting to 172.17.103.230:3306
2021-01-19 14:06:40 [INFO] FAIL
2021-01-19 14:06:40 [INFO] ======================================================================
2021-01-19 14:06:40 [INFO] FAIL: test_100_kill_crm_master (zaza.openstack.charm_tests.mysql.tests.PerconaClusterScaleTests)
2021-01-19 14:06:40 [INFO] Ensure VIP failover.
2021-01-19 14:06:40 [INFO] ----------------------------------------------------------------------
2021-01-19 14:06:40 [INFO] Traceback (most recent call last):
2021-01-19 14:06:40 [INFO]   File "/tmp/tmp.Y571dACbn5/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/mysql/tests.py", line 461, in test_100_kill_crm_master
2021-01-19 14:06:40 [INFO]     "Cannot connect to vip"
2021-01-19 14:06:40 [INFO] AssertionError: Cannot connect to vip
2021-01-19 14:06:40 [INFO] ----------------------------------------------------------------------
2021-01-19 14:06:40 [INFO] Ran 1 test in 11.920s
2021-01-19 14:06:40 [INFO] FAILED
2021-01-19 14:06:40 [INFO]  (failures=1)
2021-01-19 14:06:40 [ERROR] {'xenial_model': 'zaza-3df4042f303e'}
2021-01-19 14:06:40 [ERROR] Model xenial_model (zaza-3df4042f303e)
2021-01-19 14:06:41 [WARNING] unknown delta type: id
Traceback (most recent call last):
  File "/tmp/tmp.Y571dACbn5/func/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/tmp/tmp.Y571dACbn5/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 278, in main
    test_directory=args.test_directory)
  File "/tmp/tmp.Y571dACbn5/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 213, in func_test_runner
    force=force, test_directory=test_directory)
  File "/tmp/tmp.Y571dACbn5/func/lib/python3.5/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 149, in run_env_deployment
    test_directory=test_directory)
  File "/tmp/tmp.Y571dACbn5/func/lib/python3.5/site-packages/zaza/charm_lifecycle/test.py", line 122, in test
    run_test_list(tests)
  File "/tmp/tmp.Y571dACbn5/func/lib/python3.5/site-packages/zaza/charm_lifecycle/test.py", line 111, in run_test_list
    get_test_runners()[runner](testcase, _testcase)
  File "/tmp/tmp.Y571dACbn5/func/lib/python3.5/site-packages/zaza/charm_lifecycle/test.py", line 73, in run_unittest
    assert test_result.wasSuccessful(), "Test run failed"
AssertionError: Test run failed
ERROR: InvocationError for command /tmp/tmp.Y571dACbn5/func/bin/functest-run-suite --keep-model (exited with code 1)

[ajkavanagh]

Closed by #409