While CloudSearch was in a degraded state, xml responses were crashing boto

[cbrinker]

During a recent route53 outage, I was seeing errors coming from the lookup of a CloudSearch domain.

'import boto; boto.connect_cloudsearch().lookup("VALID_DOMAIN")'

>>> boto.connect_cloudsearch().lookup("VALID_DOMAIN")'.lookup('VALID_DOMAIN')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python/run/venv/lib/python2.7/site-packages/boto/cloudsearch/layer2.py", line 73, in lookup
    domains = self.list_domains(domain_names=[domain_name])
  File "/opt/python/run/venv/lib/python2.7/site-packages/boto/cloudsearch/layer2.py", line 54, in list_domains
    return [Domain(self.layer1, data) for data in domain_data]
  File "/opt/python/run/venv/lib/python2.7/site-packages/boto/cloudsearch/domain.py", line 82, in __init__
    self.update_from_data(data)
  File "/opt/python/run/venv/lib/python2.7/site-packages/boto/cloudsearch/domain.py", line 91, in update_from_data
    self.num_searchable_docs = data['num_searchable_docs']
KeyError: 'num_searchable_docs'

I turned up debugging levels and found that AWS was returning an XML document that didn't have a <NumSearchableDocs>NNNNN</NumSearchableDocs> element. I dug a bit deeper and found the API says that a few of the returned elements are not "required". In particular the section under "DomainStatus > NumSearchableDocs" in this pdf: http://awsdocs.s3.amazonaws.com/cloudsearch/2011-02-01/cloudsearch-dg-2011-02-01.pdf

Looking into cloudsearch/domain.py:update_from_data(), I see that many of the elements that are listed as optional are using the data['key'] syntax instead of the data,get('key', SOME_SANE_DEFAULT?) syntax. Not sure how this team would define a "sane" default?

Here is what the faulty response looked like when it came in over the wire:

<DescribeDomainsResponse xmlns="http://cloudsearch.amazonaws.com/doc/2011-02-01/">
  <DescribeDomainsResult>
    <DomainStatusList>
      <member>
        <SearchPartitionCount>7</SearchPartitionCount>
        <SearchService>
          <Arn>arn:XXXX</Arn>
          <Endpoint>search-XXX.amazonaws.com</Endpoint>
        </SearchService>
        <SearchInstanceType>search.m2.2xlarge</SearchInstanceType>
        <Created>true</Created>
        <DomainId>9999/XXXX</DomainId>
        <Processing>false</Processing>
        <SearchInstanceCount>7</SearchInstanceCount>
        <DomainName>XXXX</DomainName>
        <RequiresIndexDocuments>false</RequiresIndexDocuments>
        <Deleted>false</Deleted>
        <DocService>
          <Arn>arn:XXXX</Arn>
          <Endpoint>doc-XXX.amazonaws.com</Endpoint>
        </DocService>
      </member>
    </DomainStatusList>
  </DescribeDomainsResult>
  <ResponseMetadata>
    <RequestId>GUID</RequestId>
  </ResponseMetadata>
</DescribeDomainsResponse>