dynamic-dynamodb crash when table is deleted #126

Closed
pragnesh opened this Issue Mar 13, 2014 · 13 comments

Projects

None yet

2 participants

@pragnesh
Contributor
Traceback (most recent call last):
  File "/home/ubuntu/dynamicenv/bin/dynamic-dynamodb", line 27, in <module>
    dynamic_dynamodb.main()
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/__init__.py", line 128, in main
    check_interval=config['global']['check_interval'])
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/__init__.py", line 45, in run
    table.ensure_provisioning(table_name, table_key)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/core/table.py", line 26, in ensure_provisioning
    table_name, key_name)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/core/table.py", line 54, in __ensure_provisioning_reads
    updated_read_units = dynamodb.get_provisioned_table_read_units(table_name)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/core/dynamodb.py", line 149, in get_provisioned_table_read_units
    desc = DYNAMODB_CONNECTION.describe_table(table_name)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 539, in describe_table
    body=json.dumps(params))
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 1483, in make_request
    retry_handler=self._retry_handler)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/boto/connection.py", line 932, in _mexe
    status = retry_handler(response, i, next_sleep)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 1526, in _retry_handler
    data)
boto.exception.JSONResponseError: JSONResponseError: 400 Bad Request
{u'message': u'Requested resource not found: Table: f23_83 not found', u'__type': u'com.amazonaws.dynamodb.v20120810#ResourceNotFoundException'}
@pragnesh
Contributor

started seeing above exception when table gets deleted, and dyanmic-dynamodb stopped working, i am using latest release 1.9.0

@sebdah
Owner
sebdah commented Mar 13, 2014

Thanks for the report @pragnesh, I will have a look at catching this error.

@sebdah sebdah added this to the 1.9.x milestone Mar 13, 2014
@sebdah sebdah self-assigned this Mar 13, 2014
@pragnesh
Contributor

i am also seeing that sometime "newly created table" are not managed even if matches table regular expression

@sebdah
Owner
sebdah commented Mar 13, 2014

Ok, thanks.

I have started implementing a fix for this. Could you please test this in your environment? I have released an alpha release to PyPI:

pip install dynamic-dynamodb==1.9.1a1
@pragnesh
Contributor

i have deployed this release in my environment, i will let you know in a day wheather everything is fine or not, thanks for fix

@sebdah
Owner
sebdah commented Mar 13, 2014

Fantastic, thank you!
On Mar 13, 2014 4:35 PM, "pragnesh" notifications@github.com wrote:

i have deployed this release in my environment, i will let you in a day
wheather everything is fine or not

Reply to this email directly or view it on GitHubhttps://github.com/sebdah/dynamic-dynamodb/issues/126#issuecomment-37547408
.

@pragnesh
Contributor

now i see following line in log and dynamic-dynamodb stopped working

2014-03-14 02:33:41,623 - dynamic-dynamodb - ERROR - f17_253 - Table f17_253 does not exist anymore

@sebdah
Owner
sebdah commented Mar 14, 2014

Thanks for the report @pragnesh. I think I can see the reason and I have moved the exception handling to a better position. Can you please test again?

pip install dynamic-dynamodb==1.9.1a2

Have a nice weekend

@pragnesh
Contributor

thanks for fix , i have deployed new release in our env.

@pragnesh
Contributor

this seems to working fine now, but it appears to me that by looking at code that daemon mode no longer update table list from aws, but non daemon mode does update.

This behavior is different from earlier release. This may create problem when table is deleted or regex is used so there may be new table available to manage.

For the time being i switch to non daemon mode which work for us with supervisor.

@sebdah
Owner
sebdah commented Mar 17, 2014

Thanks for all the testing @pragnesh!

The problem with the daemon is not introduced by this release, and I think that is the answer to your comment above.

Are you in a rush to get the fix with the deleted table out? If not I'll see if I can fix the issue with the daemon mode and stall table list before releasing the hotfix.

@sebdah
Owner
sebdah commented Mar 17, 2014

I have created a new issue for dealing with the daemon mode issue (#127)

@sebdah sebdah added a commit that referenced this issue Mar 17, 2014
@sebdah Added release notes for #126 f792283
@sebdah sebdah closed this Mar 17, 2014
@sebdah
Owner
sebdah commented Mar 17, 2014

This has been fixed in version 1.9.1, thanks for the report @pragnesh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment