Beaker provision failing again due to 'Ansible Undefined' issue

[Dannyb48]

Describe the bug
Carbon is failing to provision Beaker resources again on Linchpin develop branch. The error Carbon throws is

2020-01-10 22:43:40,579 ERROR [carbon.tasks.provision.run:76] Failed to provision asset db2-carbon-beaker-node
2020-01-10 22:43:40,580 ERROR [carbon.tasks.provision.run:78] string indices must be integers
Traceback (most recent call last):
  File "/home/jenkins/workspace/e2e-beaker-linchpin-wrapper-2/cbn/lib/python2.7/site-packages/blaster/engine/serial.py", line 42, in run
    value = getattr(task_obj, method)()
  File "/home/jenkins/workspace/e2e-beaker-linchpin-wrapper-2/cbn/lib/python2.7/site-packages/carbon/tasks/provision.py", line 74, in run
    return self.provisioner.create()
  File "/home/jenkins/workspace/e2e-beaker-linchpin-wrapper-2/cbn/lib/python2.7/site-packages/carbon/provisioners/ext/linchpin_wrapper_plugin.py", line 376, in create
    res = self._create()
  File "/home/jenkins/workspace/e2e-beaker-linchpin-wrapper-2/cbn/lib/python2.7/site-packages/carbon/provisioners/ext/linchpin_wrapper_plugin.py", line 305, in _create
    res.append({'ip': str(lookup_ip_of_hostname(bkr_server['system'])),
TypeError: string indices must be integers

But I think that is because the Ansible Undefined issue is back. I see the following

'resource_group': AnsibleUndefined

The results of the provision shows it here:

{"1": {"action": "up", "summary_data": {"carbon": {"1": {"uhash": "9970ed", "rc": 0}}}, "results_data": {"carbon": {"task_results": null, "rundb_data": {"1": {"inputs": [{"topology_data": {"topology_name": "carbon", "resource_groups": [{"resource_group_name": "carbon", "resource_definitions": [{"ssh_keys_path": "/home/jenkins/workspace/e2e-beaker-linchpin-wrapper-2/keys", "role": "bkr_server", "whiteboard": "CCIT Carbon E2E Acceptance Testing", "job_group": "****", "recipesets": [{"count": 1, "name": "db2-carbon-beaker-node", "variant": "Server", "hostrequires": [{"force": "dell-per220-02.cci.lab.eng.bos.redhat.com"}], "ssh_key_file": ["demo.pub"], "arch": "x86_64", "distro": "RHEL-7.6"}]}], "resource_group_type": "beaker"}]}}, {"layout_data": {"inventory_layout": {"hosts": [{"count": 1, "name": "node", "host_groups": ["hosts", "client"]}], "vars": {"hostname": "db2-carbon-beaker-node", "ansible_ssh_private_key_file": "keys/demo", "ansible_user": "****"}}}}], "end": "1578696220.5", "cfgs": [], "rc": 0, "uhash": "9970ed", "start": "1578695063.97", "hooks": [], "action": "up", "outputs": [{"bkr_id_values": [{"url": "https://beaker.engineering.redhat.com/jobs/4007711", "id": "4007711"}]}, {"resources": "[{u'status': u'Waiting', 'name': u'db2-carbon-beaker-node', u'family': u'RedHatEnterpriseLinux7', 'url': u'https://beaker.engineering.redhat.com/jobs/4007711', 'resource_group': AnsibleUndefined, u'variant': u'Server', u'system': u'dell-per220-02.cci.lab.eng.bos.redhat.com', 'role': u'bkr_server', u'result': u'Pass', u'rid': u'7783111', u'arch': u'x86_64', u'id': u'4007711', u'distro': u'RHEL-7.6'}]"}, {"inventory_path": ["/home/jenkins/workspace/e2e-beaker-linchpin-wrapper-2/.carbon/.results/linchpin/inventories/carbon.inventory"]}]}}}}}}

To Reproduce
Steps to reproduce the behavior:

1. using python2 virtualenv
2. pip install develop branch of carbon
3. pip install develop branch of linchpin

You can ping me on where you can grab an example carbon scenario provisioning beaker with linchpin

[Dannyb48]

@14rcole any chance you can take a look at this? You fixed this last time? Does this have anything to do with any recent changes with turning the roles into ansible galaxy roles?

[14rcole]

@Dannyb48 Yeah, I'll take a look. There weren't any recent changes to the ansible galaxy roles or to the beaker role specifically

[Dannyb48]

FYI this is a blocker for us, without this fix we can't support the next version of Linchpin