Skip to content

Conversation

juliusgeo
Copy link
Contributor

No description provided.

@juliusgeo juliusgeo requested a review from ShaneHarvey October 7, 2021 17:36
if client._get_topology._closed:
return False
else:
raise e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this will actually work. The problem is that _process_periodic_tasks already catches and logs the InvalidOperation error which is want we want to prevent. We need to refactor this kind of logic so that we can ignore InvalidOperation without logging it at all:

            try:
                self._cleanup_cursor(True, cursor_id, address, sock_mgr,
                                     None, False)
            except Exception:
                helpers._handle_exception()

It would also be good to add a test for this. Perhaps something like this would work but it requires some experimentation:

coll.insert_many([{} for _ in range(5])
cursor = coll.find(batchSize=2)
cursor.next()
assert cursor.cursor_id
client.close()
# Add cursor to kill cursors queue 
del cursor
wait_until(cursor_id is added to client's kill cursors queue)
client._process_periodic_tasks()   # This must not raise or print any exceptions

@juliusgeo juliusgeo requested a review from ShaneHarvey October 7, 2021 22:20
client.close()
# Add cursor to kill cursors queue
del cursor
wait_until(lambda: c_id in [c for _, c, _ in
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can simplify this to lambda: client._MongoClient__kill_cursors_queue

client._process_periodic_tasks() # This must not raise or print any exceptions
except Exception:
self.fail("client._process_periodic_tasks() raised an exception")


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need the try/except/fail here. Just call client._process_periodic_tasks(). If that raises unexpectedly the teat will fail automatically.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

cursor = coll.find(batch_size=2)
cursor.next()
c_id = cursor.cursor_id
assert c_id
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should use self.assertIsNotNone here.

@@ -1591,6 +1591,25 @@ def test_network_error_message(self):
with self.assertRaisesRegex(AutoReconnect, expected):
client.pymongo_test.test.find_one({})

def test_process_periodic_tasks(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this test will fail on PyPy because garbage collection works slightly differently. Let's skip this test entirely when running with PyPy (see my current PR for how to do that)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -1583,6 +1583,8 @@ def _process_kill_cursors(self):
try:
self._cleanup_cursor(True, cursor_id, address, sock_mgr,
None, False)
except InvalidOperation as e:
raise e
except Exception:
helpers._handle_exception()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's another try/except thats needs to be updated below too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's still missing (line 1602).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

self._topology.update_pool(self.__all_credentials)
except InvalidOperation:
pass
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only want to ignore this error if the client was closed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@ShaneHarvey
Copy link
Member

We need to fix this bug:

 [2021/10/08 02:08:52.536] ERROR: MongoClient has no attribute '_MongoClient__kill_cursors_queuee'. To access the _MongoClient__kill_cursors_queuee database, use client['_MongoClient__kill_cursors_queuee']. (AttributeError)
 [2021/10/08 02:08:52.536] Traceback (most recent call last):
 [2021/10/08 02:08:52.536]   File "/data/mci/4133ce1e64692bb1a00e7fbf98b2478b/src/test/test_client.py", line 1607, in test_process_periodic_tasks
 [2021/10/08 02:08:52.536]     "waited for cursor to be added to queue")
 [2021/10/08 02:08:52.536]   File "/data/mci/4133ce1e64692bb1a00e7fbf98b2478b/src/test/utils.py", line 750, in wait_until
 [2021/10/08 02:08:52.536]     retval = predicate()
 [2021/10/08 02:08:52.536]   File "/data/mci/4133ce1e64692bb1a00e7fbf98b2478b/src/test/test_client.py", line 1606, in <lambda>
 [2021/10/08 02:08:52.536]     wait_until(lambda: client._MongoClient__kill_cursors_queuee,
 [2021/10/08 02:08:52.536]   File "/data/mci/4133ce1e64692bb1a00e7fbf98b2478b/src/pymongo/mongo_client.py", line 1469, in __getattr__
 [2021/10/08 02:08:52.536]     " database, use client[%r]." % (name, name, name))
 [2021/10/08 02:08:52.536] AttributeError: MongoClient has no attribute '_MongoClient__kill_cursors_queuee'. To access the _MongoClient__kill_cursors_queuee database, use client['_MongoClient__kill_cursors_queuee'].

@@ -1583,6 +1583,11 @@ def _process_kill_cursors(self):
try:
self._cleanup_cursor(True, cursor_id, address, sock_mgr,
None, False)
except InvalidOperation as e:
if self._topology._closed:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth adding a comment here to explain why we're raising the error when the client is closed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -1583,6 +1583,11 @@ def _process_kill_cursors(self):
try:
self._cleanup_cursor(True, cursor_id, address, sock_mgr,
None, False)
except InvalidOperation as e:
if self._topology._closed:
raise e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should use bare raise instead of raise e.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

if self._topology._closed:
pass
else:
helpers._handle_exception()
except Exception:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be cleaner:

except Exception as exc:
    if isinstance(exc, InvalidOperation) and self._topology._closed:
        return # Ignore when client has been closed. 

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -1583,6 +1583,8 @@ def _process_kill_cursors(self):
try:
self._cleanup_cursor(True, cursor_id, address, sock_mgr,
None, False)
except InvalidOperation as e:
raise e
except Exception:
helpers._handle_exception()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's still missing (line 1602).

@juliusgeo
Copy link
Contributor Author

We need to fix this bug:

 [2021/10/08 02:08:52.536] ERROR: MongoClient has no attribute '_MongoClient__kill_cursors_queuee'. To access the _MongoClient__kill_cursors_queuee database, use client['_MongoClient__kill_cursors_queuee']. (AttributeError)
 [2021/10/08 02:08:52.536] Traceback (most recent call last):
 [2021/10/08 02:08:52.536]   File "/data/mci/4133ce1e64692bb1a00e7fbf98b2478b/src/test/test_client.py", line 1607, in test_process_periodic_tasks
 [2021/10/08 02:08:52.536]     "waited for cursor to be added to queue")
 [2021/10/08 02:08:52.536]   File "/data/mci/4133ce1e64692bb1a00e7fbf98b2478b/src/test/utils.py", line 750, in wait_until
 [2021/10/08 02:08:52.536]     retval = predicate()
 [2021/10/08 02:08:52.536]   File "/data/mci/4133ce1e64692bb1a00e7fbf98b2478b/src/test/test_client.py", line 1606, in <lambda>
 [2021/10/08 02:08:52.536]     wait_until(lambda: client._MongoClient__kill_cursors_queuee,
 [2021/10/08 02:08:52.536]   File "/data/mci/4133ce1e64692bb1a00e7fbf98b2478b/src/pymongo/mongo_client.py", line 1469, in __getattr__
 [2021/10/08 02:08:52.536]     " database, use client[%r]." % (name, name, name))
 [2021/10/08 02:08:52.536] AttributeError: MongoClient has no attribute '_MongoClient__kill_cursors_queuee'. To access the _MongoClient__kill_cursors_queuee database, use client['_MongoClient__kill_cursors_queuee'].

This typo has been fixed.

Copy link
Member

@ShaneHarvey ShaneHarvey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two minor comments

@@ -1607,17 +1612,24 @@ def _process_kill_cursors(self):
self._kill_cursors(
cursor_ids, address, topology, session=None)
except Exception:
helpers._handle_exception()
if (isinstance(e,InvalidOperation) and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a space between e, and InvalidOperation.

@@ -1596,8 +1596,13 @@ def _process_kill_cursors(self):
try:
self._cleanup_cursor(True, cursor_id, address, sock_mgr,
None, False)
except Exception:
helpers._handle_exception()
except Exception as e:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rename e to exc everywhere. We usually use exc as the name for exceptions.

Copy link
Member

@ShaneHarvey ShaneHarvey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@juliusgeo juliusgeo merged commit a4ccfa5 into mongodb:master Oct 9, 2021
juliusgeo added a commit to juliusgeo/mongo-python-driver that referenced this pull request Apr 5, 2022
juliusgeo added a commit to juliusgeo/mongo-python-driver that referenced this pull request Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants