GCE ex_list_instancegroups('all') raises KeyError 'zone'

[kartiksubbarao]

Summary

The GCE ex_list_instancegroups('all') call raises KeyError 'zone'.

Detailed Information

Libcloud 2.5.0
Python 3.5
Ubuntu 16.04.6 LTS

The error is caused by line 9092 of gce.py in the _to_instancegroup() function:

zone = self.ex_get_zone(instancegroup['zone'])

The code is expecting the instancegroup dict to have a zone attribute, but that's not always the case. The underlying GCE API endpoint /aggregated/instanceGroups can also return objects that have a region attribute but not a zone attribute. Example:

{'creationTimestamp': '2019-05-20T12:00:00.000-08:00',
 'description': 'This instance group is controlled by Regional Instance Group '
                "Manager 'abcd-ef-ghij'. To modify instances in this group, "
                'use the Regional Instance Group Manager API: '
                'https://cloud.google.com/compute/docs/reference/latest/instanceGroupManagers',
 'fingerprint': 'AbCdEfGHiJK=',
 'id': '12345678901234567',
 'kind': 'compute#instanceGroup',
 'name': 'abcd-ef-ghij',
 'namedPorts': [{'name': 'https', 'port': 443}],
 'network': 'https://www.googleapis.com/compute/v1/projects/testproj1/global/networks/ab',
 'region': 'https://www.googleapis.com/compute/v1/projects/testproj1/regions/us-west1',
 'selfLink': 'https://www.googleapis.com/compute/v1/projects/testproj1/regions/us-west1/instanceGroups/abcd-ef-ghij',
 'size': 0,
 'subnetwork': 'https://www.googleapis.com/compute/v1/projects/testproj1/regions/us-west1/subnetworks/asdf1234'}

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.

[Kami]

@kartiksubbarao Sorry for a late response.

Is this still an issue? I tried, but can't reproduce the issue with the latest version of Libcloud.

In any case, we could probably still do something like zone = self.ex_get_zone(instancegroup['zone']) if instancegroup.get('zone, None) else None

I'm also wondering why zone attribute is not populated / present.

[kartiksubbarao]

Thanks for the response @Kami . I don't have access to that GCE environment anymore so I can't test, but I compared the August libcloud code to the current code and it doesn't seem like there were any changes to libcloud/compute/drivers/gce.py that would have fixed this issue:

5cbd788...4107267

Hopefully the example object I provided above might shed some light into how GCP can return objects that have regions (but not zones) populated.

Also, here is some code that I came up with to work around the issue, perhaps this may help in developing a proper fix:

def list_all_instancegroups(gcp, projectid):
    instance_groups = []
    resp = gcp.connection.request('/aggregated/instanceGroups').object
    if 'items' in resp:
        for v in resp['items'].values():
            for ig in v.get('instanceGroups', []):
                if 'zone' in ig:
                    try: instance_groups.append(gcp._to_instancegroup(ig))
                    except:
                        e = sys.exc_info()
                        print('{}: instancegroup: {}'.format(
                             projectid, e[1].value['message']), file=sys.stderr)
                        if e[0] != libcloud.common.google.ResourceNotFoundError:
                            raise
    return instance_groups

[Kami]

Thanks for confirming.

I'm not exactly sure why that attribute is not present and when I that can happen, but I pushed a change in facae52 which I believe should solve that issue (?) and I saw similar pattern is already used for other attributes.

[kartiksubbarao]

@Kami I did some further searching and came across this:

https://cloud.google.com/compute/docs/instance-groups/distributing-instances-with-regional-instance-groups

It looks like GCE supports creating instance groups that are associated with all of the zones in a particular region. In this case, it's the region that's the geographic defining attribute, not an individual zone. That's why the zone attribute isn't there -- the region attribute is used instead.