Cloudsearch: 400 Bad Request - Rate Exceeded #1976

Open
MattTheRed opened this Issue Jan 9, 2014 · 1 comment

Comments

Projects
None yet
2 participants

When using boto to connect to CloudSearch, we sometimes get throttle limit errors from amazon from the get_document_service() call.

    domain = cs_conn.lookup("domain-name")
    document_service = domain.get_document_service()
    search_service = document_service.get_search_service()
    results = search_service.search(**search_kwargs)
<ErrorResponse xmlns=”http://cloudsearch.amazonaws.com/doc/2011-02-01/">
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>
  <RequestId>c753f49e-26b9-11e3-a879-0d3b8f7ade79</RequestId>
</ErrorResponse>

We've handled it by just caching the search service, but that might be something better handled in the boto library.

import boto
from django.core.cache import cache as django_cache

cs_conn = boto.connect_cloudsearch(
    aws_access_key_id=YOUR_KEY_ID,
    aws_secret_access_key=YOUR_ACCESS_KEY,
)

search_kwargs = {}
search_kwargs['q'] = "Stuff to find"

try:
    search_service = django_cache.get("cloudsearch_search_service")
    # Need to try to execute something for it to fail         
    results = search_service.search(**search_kwargs)
except:
    domain = cs_conn.lookup("domain-name")
    document_service = domain.get_document_service()
    search_service = document_service.get_search_service()
    django_cache.set("cloudsearch_search_service", search_service, 300)
    results = search_service.search(**search_kwargs)

print results.doc

Hi MattTheRed,

I understood your solution. But on my scenario we need to know the real index status over specific cloudsearch domain, we can not cache thar result.
Any other solution?
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment