New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dynamic-dynamodb crash when table is deleted #126

Closed
pragnesh opened this Issue Mar 13, 2014 · 13 comments

Comments

Projects
None yet
2 participants
@pragnesh
Contributor

pragnesh commented Mar 13, 2014

Traceback (most recent call last):
  File "/home/ubuntu/dynamicenv/bin/dynamic-dynamodb", line 27, in <module>
    dynamic_dynamodb.main()
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/__init__.py", line 128, in main
    check_interval=config['global']['check_interval'])
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/__init__.py", line 45, in run
    table.ensure_provisioning(table_name, table_key)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/core/table.py", line 26, in ensure_provisioning
    table_name, key_name)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/core/table.py", line 54, in __ensure_provisioning_reads
    updated_read_units = dynamodb.get_provisioned_table_read_units(table_name)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/dynamic_dynamodb/core/dynamodb.py", line 149, in get_provisioned_table_read_units
    desc = DYNAMODB_CONNECTION.describe_table(table_name)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 539, in describe_table
    body=json.dumps(params))
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 1483, in make_request
    retry_handler=self._retry_handler)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/boto/connection.py", line 932, in _mexe
    status = retry_handler(response, i, next_sleep)
  File "/home/ubuntu/dynamicenv/local/lib/python2.7/site-packages/boto/dynamodb2/layer1.py", line 1526, in _retry_handler
    data)
boto.exception.JSONResponseError: JSONResponseError: 400 Bad Request
{u'message': u'Requested resource not found: Table: f23_83 not found', u'__type': u'com.amazonaws.dynamodb.v20120810#ResourceNotFoundException'}
@pragnesh

This comment has been minimized.

Contributor

pragnesh commented Mar 13, 2014

started seeing above exception when table gets deleted, and dyanmic-dynamodb stopped working, i am using latest release 1.9.0

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 13, 2014

Thanks for the report @pragnesh, I will have a look at catching this error.

@sebdah sebdah added this to the 1.9.x milestone Mar 13, 2014

@sebdah sebdah self-assigned this Mar 13, 2014

@pragnesh

This comment has been minimized.

Contributor

pragnesh commented Mar 13, 2014

i am also seeing that sometime "newly created table" are not managed even if matches table regular expression

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 13, 2014

Ok, thanks.

I have started implementing a fix for this. Could you please test this in your environment? I have released an alpha release to PyPI:

pip install dynamic-dynamodb==1.9.1a1
@pragnesh

This comment has been minimized.

Contributor

pragnesh commented Mar 13, 2014

i have deployed this release in my environment, i will let you know in a day wheather everything is fine or not, thanks for fix

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 13, 2014

Fantastic, thank you!
On Mar 13, 2014 4:35 PM, "pragnesh" notifications@github.com wrote:

i have deployed this release in my environment, i will let you in a day
wheather everything is fine or not

Reply to this email directly or view it on GitHubhttps://github.com//issues/126#issuecomment-37547408
.

@pragnesh

This comment has been minimized.

Contributor

pragnesh commented Mar 14, 2014

now i see following line in log and dynamic-dynamodb stopped working

2014-03-14 02:33:41,623 - dynamic-dynamodb - ERROR - f17_253 - Table f17_253 does not exist anymore

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 14, 2014

Thanks for the report @pragnesh. I think I can see the reason and I have moved the exception handling to a better position. Can you please test again?

pip install dynamic-dynamodb==1.9.1a2

Have a nice weekend

@pragnesh

This comment has been minimized.

Contributor

pragnesh commented Mar 14, 2014

thanks for fix , i have deployed new release in our env.

@pragnesh

This comment has been minimized.

Contributor

pragnesh commented Mar 17, 2014

this seems to working fine now, but it appears to me that by looking at code that daemon mode no longer update table list from aws, but non daemon mode does update.

This behavior is different from earlier release. This may create problem when table is deleted or regex is used so there may be new table available to manage.

For the time being i switch to non daemon mode which work for us with supervisor.

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 17, 2014

Thanks for all the testing @pragnesh!

The problem with the daemon is not introduced by this release, and I think that is the answer to your comment above.

Are you in a rush to get the fix with the deleted table out? If not I'll see if I can fix the issue with the daemon mode and stall table list before releasing the hotfix.

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 17, 2014

I have created a new issue for dealing with the daemon mode issue (#127)

sebdah added a commit that referenced this issue Mar 17, 2014

@sebdah sebdah closed this Mar 17, 2014

@sebdah

This comment has been minimized.

Owner

sebdah commented Mar 17, 2014

This has been fixed in version 1.9.1, thanks for the report @pragnesh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment