Boto error in a replay

[ggreg]

11/14/2014 02:30:03 ERROR:Cannot replay the workflow TypeError(string indices must be integers, not str)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/simpleflow/swf/executor.py", line 295, in replay
    result = self.run_workflow(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/simpleflow/executor.py", line 59, in run_workflow
    return self._workflow.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cdf/workflows/analysis.py", line 555, in run
    for part_id in partitions.result
  File "/usr/local/lib/python2.7/dist-packages/simpleflow/workflow.py", line 30, in submit
    return self._executor.submit(func, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/simpleflow/swf/executor.py", line 261, in submit
    return self.resume(task, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/simpleflow/swf/executor.py", line 232, in resume
    self.schedule_task(task)
  File "/usr/local/lib/python2.7/dist-packages/simpleflow/swf/executor.py", line 201, in schedule_task
    decisions = task.schedule(self.domain)
  File "/usr/local/lib/python2.7/dist-packages/simpleflow/swf/task.py", line 14, in schedule
    version=activity.version,
  File "/usr/local/lib/python2.7/dist-packages/swf/utils.py", line 159, in __new__
    new = mutableclass(*args, **kw) # __init__ gets called while still mutable
  File "/usr/local/lib/python2.7/dist-packages/swf/models/activity.py", line 121, in __init__
    super(self.__class__, self).__init__(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/swf/core.py", line 40, in __init__
    boto.swf.connect_to_region(self.region, **settings_))
  File "/usr/local/lib/python2.7/dist-packages/boto/swf/__init__.py", line 45, in connect_to_region
    return region.connect(**kw_params)
  File "/usr/local/lib/python2.7/dist-packages/boto/regioninfo.py", line 188, in connect
    return self.connection_cls(region=self, **kw_params)
  File "/usr/local/lib/python2.7/dist-packages/boto/swf/layer1.py", line 85, in __init__
    debug, session_token, profile_name=profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 558, in __init__
    profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 197, in __init__
    self.get_credentials(access_key, secret_key, security_token, profile_name)
  File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 365, in get_credentials
    self._populate_keys_from_metadata_server()
  File "/usr/local/lib/python2.7/dist-packages/boto/provider.py", line 384, in _populate_keys_from_metadata_server
    self._access_key = security['AccessKeyId']
TypeError: string indices must be integers, not str

[jbbarth]

Happens again today. It seems it happens when many decisions are taken at once.

Useful informations: http://blog.johnryding.com/ + http://blog.gmane.org/gmane.comp.python.boto/month=20150701

I didn't analyze myself, but if I believe what's told in those articles, the retrieval of temporary credentials can't keep up under load. So either centralize things (to avoid potentially that 100+ decider processes ask the same API in a short period of time) either add some retries to avoid that.

Also we could definitely bake our own retry logic inside swf/core.py, that's probably even simpler.

[ybastide]

(First (perma)link: http://blog.johnryding.com/post/122337566993/solving-intermittent-noauthhandlerfound-errors-in)

A singleton Provider looks only useful in standalone use, so we may want to use @retry as you say

[jbbarth]

Fixed in #99 / 0.11.8