Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

World: high-yield queries failing in production #3

Closed
toconnell opened this issue Jul 23, 2019 · 7 comments

Comments

@toconnell
Copy link
Contributor

commented Jul 23, 2019

Interestingly, it works in dev (using the production database), but fails in production thus:

[2019-07-23 13:43:51] ERROR:    Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.
Traceback (most recent call last):
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 160, in refresh_asset
    self.update_asset_dict(asset_dict)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 214, in update_asset_dict
    fresh_results = update_method()
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 836, in latest_survivor
    s = self.get_eligible_documents(collection="survivors", limit=1, include_settlement=True)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 442, in get_eligible_documents
    results = [x for x in results]
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 442, in <listcomp>
    results = [x for x in results]
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 1189, in next
    if len(self.__data) or self._refresh():
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 1104, in _refresh
    self.__send_message(q)
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 982, in __send_message
    helpers._check_command_response(first)
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/helpers.py", line 155, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.
@toconnell toconnell self-assigned this Jul 23, 2019
@toconnell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 23, 2019

This is almost certainly a problem in get_eligible_documents() (in the world app) with the way we had to implement limit() using sort due to limitations of the old-ass version of pymongo used before the migration.

@toconnell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 23, 2019

This has been optimized to use the built-in limit() method, instead of how we used to do it. I also moved the list comprehension to the end of the get_eligible_documents() method.

@toconnell toconnell closed this Jul 23, 2019
@toconnell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 23, 2019

This is continuing to happen on large sets of results:

[2019-07-23 16:45:52] ERROR:    exception caught while refreshing 'avg_fighting_arts' asset!
[2019-07-23 16:45:52] ERROR:    Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.
Traceback (most recent call last):
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 165, in refresh_asset
    self.update_asset_dict(asset_dict)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 219, in update_asset_dict
    fresh_results = update_method()
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 792, in avg_fighting_arts
    return self.get_average("survivors", "fighting_arts")
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 499, in get_average
    sample_set = self.get_eligible_documents(collection, attrib)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 459, in get_eligible_documents
    results = [x for x in results]
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 459, in <listcomp>
    results = [x for x in results]
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 1189, in next
    if len(self.__data) or self._refresh():
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 1104, in _refresh
    self.__send_message(q)
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 982, in __send_message
    helpers._check_command_response(first)
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/helpers.py", line 155, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.
@toconnell toconnell reopened this Jul 23, 2019
@toconnell toconnell changed the title World: 'latest_survivor' query is failing in production World: high-yield queries failing in production Jul 24, 2019
@toconnell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 24, 2019

Doing results = list(results) instead of results - [x for x in results] does not fix the problem.

@toconnell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 24, 2019

Added the create_indexes() method to the core world module. Created indexes for survivors and settlements that should help us stream these results.

Looks good in test; still have to try it in production and see if it stops the pymongo errors.

@toconnell toconnell added the bug label Jul 25, 2019
@toconnell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2019

Added indexes in production:

[2019-07-25 17:52:24] WARNING:  Creating indexes for 'survivors' collection!
[2019-07-25 17:52:25] INFO:     [1] survivors index: SON([('v', 2), ('key', SON([('_id', 1)])), ('name', '_id_'), ('ns', 'kdm-manager.survivors')])
[2019-07-25 17:52:25] INFO:     [2] survivors index: SON([('v', 2), ('unique', True), ('key', SON([('created_by', 1), ('created_on', -1)])), ('name', 'created_by_1_created_on_-1'), ('ns', 'kdm-manager.survivors')])
[2019-07-25 17:52:25] INFO:     [3] survivors index: SON([('v', 2), ('key', SON([('created_by', 1), ('created_on', -1), ('settlement', 1)])), ('name', 'created_by_1_created_on_-1_settlement_1'), ('ns', 'kdm-manager.survivors')])
[2019-07-25 17:52:25] WARNING:  Creating indexes for 'settlements' collection!
[2019-07-25 17:52:25] INFO:     [1] settlements index: SON([('v', 2), ('key', SON([('_id', 1)])), ('name', '_id_'), ('ns', 'kdm-manager.settlements')])
[2019-07-25 17:52:25] INFO:     [2] settlements index: SON([('v', 2), ('unique', True), ('key', SON([('created_by', 1), ('created_on', -1)])), ('name', 'created_by_1_created_on_-1'), ('ns', 'kdm-manager.settlements')])
[2019-07-25 17:52:25] WARNING:  Creating indexes for 'settlement_events' collection!
[2019-07-25 17:52:29] INFO:     [1] settlement_events index: SON([('v', 2), ('key', SON([('_id', 1)])), ('name', '_id_'), ('ns', 'kdm-manager.settlement_events')])
[2019-07-25 17:52:29] INFO:     [2] settlement_events index: SON([('v', 2), ('key', SON([('settlement_id', 1), ('created_by', 1), ('created_on', -1)])), ('name', 'settlement_id_1_created_by_1_created_on_-1'), ('ns', 'kdm-manager.settlement_events')])

Still failing:

toconnell@advanced-kdm-manager:~/kdm-manager-api$ ./world.sh --debug avg_fighting_arts
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "app/world/__main__.py", line 75, in <module>
    results = WORLD.debug_query(options.debug_query)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 400, in debug_query
    results = self.update_asset_dict(asset_dict)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 271, in update_asset_dict
    fresh_results = update_method()
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 843, in avg_fighting_arts
    return self.get_average("survivors", "fighting_arts")
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 550, in get_average
    sample_set = self.get_eligible_documents(collection, attrib)
  File "/home/toconnell/kdm-manager-api/app/utils/__init__.py", line 239, in not_timed
    return method(*args, **kwargs)
  File "/home/toconnell/kdm-manager-api/app/world/__init__.py", line 510, in get_eligible_documents
    results = list(results)
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 1189, in next
    if len(self.__data) or self._refresh():
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 1104, in _refresh
    self.__send_message(q)
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/cursor.py", line 982, in __send_message
    helpers._check_command_response(first)
  File "/home/toconnell/kdm-manager-api/venv/lib/python3.6/site-packages/pymongo/helpers.py", line 155, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a s
maller limit.
@toconnell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 26, 2019

OK, the solution that looks like the winner here is to try it and, if we fail, try it again sorted by the user (i.e. created_by).

I'm calling this one resolved for now.

@toconnell toconnell closed this Jul 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.