bpo-26543: Fix IMAP4.noop when debug mode is enabled #15206

CuriousLearner · 2019-08-11T11:08:31Z

This supersedes #1503

I've tried to debug this issue and have the following observations:

In case if dump_ur is directly called, then with this new patch, we're gracefully able to handle the bytes data and the new test cases pass.
In case when simple_command is directly for noop, the self.get_response call inside _get_tagged_response raises an error:

socket error: unterminated line: b'* CAPABILITY IMAP4rev1 AUTH\n'

It is worth noting that this test case fails even if the debug level is decremented from the original of 3 as described in the bug.

I'm not sure what to do in such a situation to gracefully handle this.

In case I just return the typ and data as in https://github.com/python/cpython/compare/master...CuriousLearner:[bpo-26543](https://bugs.python.org/issue26543)?expand=1#diff-ed3421b57957b3ffd24b79e746a88db4R1023

The test will pass, but I'm not sure what is the best approach to take here. Can you please provide a pointer?

https://bugs.python.org/issue26543

CuriousLearner · 2019-08-11T11:09:07Z

Hi @vstinner

Can you take a look at this and questions I posted above?

Thanks!

maxking

I think the fix for _dump_ur should work, but we need a test case to make sure that it does indeed handle bytes data.

We don't necessarily need to boot up a server, we could just pass the expected response data dict to the method and see that it prints correctly (we can capture stderr, so that it doesn't pollute test output).

maxking · 2019-10-06T19:33:33Z

Lib/imaplib.py

+            if name == "NOOP":
+                return typ, data


I don't think this is needed, if there are no untagged response from the server, otherwise, the command error is expected.

maxking · 2019-10-06T19:35:34Z

Lib/test/test_imaplib.py

@@ -879,6 +879,14 @@ def test_with_statement_logout(self):
                self.assertIsNone(server.logged)
            self.assertIsNone(server.logged)

+    @reap_threads
+    def test_noop_with_debug(self):


For this test to work, there needs to be some untagged response from the server.

A simpler test case IMO, which can reproduce the issue is to have a simple untagged_response_dict passed directly to _dump_ur and make sure that it is correctly able to handle bytes data.

Yeah, I concur: test directly _dump_ur(). But since it's a private method, please decorate the test with @support.cpython_only since _dump_ur() is an implementation detail.

bedevere-bot · 2019-10-06T19:37:58Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

vstinner · 2019-10-07T07:11:48Z

Lib/imaplib.py

+        def _dump_ur(self, untagged_resp_dict):
+            if untagged_resp_dict:
+                self._mesg('untagged responses dump:' +
+                    ''.join('\n\t\t%s: %r' % (x[0], x[1]) for x in untagged_resp_dict.items()))


I propose:

def _dump_ur(self, untagged_resp_dict): if not untagged_resp_dict: return items = ('%s: %r' % (key, value) for key, value in untagged_resp_dict.items()) self._mesg('untagged responses dump:' + '\n\t\t'.join(items))

vstinner · 2019-10-07T07:13:45Z

Lib/test/test_imaplib.py

@@ -879,6 +879,14 @@ def test_with_statement_logout(self):
                self.assertIsNone(server.logged)
            self.assertIsNone(server.logged)

+    @reap_threads
+    def test_noop_with_debug(self):


Yeah, I concur: test directly _dump_ur(). But since it's a private method, please decorate the test with @support.cpython_only since _dump_ur() is an implementation detail.

vstinner · 2019-10-07T07:15:58Z

Misc/NEWS.d/next/Library/2019-08-11-16-28-03.bpo-26543.X-TJZO.rst

@@ -0,0 +1 @@
+Fix crash on :meth:`IMAP4.noop()` when bytes data is passed.


For me, a "crash" means that the Python process is killed immediately, like a SIGSEGV. Here it's "just" a regular exception. Moreover, you should mention that the issue only affects debug mode. I suggest:

Fix :meth:`IMAP4.noop()` when debug mode is enabled (ex: ``imaplib.Debug = 3``).

csabella · 2020-01-12T13:40:43Z

@CuriousLearner, please address the review comments from Victor. Thanks!

rotuna · 2020-04-23T23:17:01Z

I've implemented all requested changes in another PR (#19693)

CuriousLearner · 2020-04-27T11:28:07Z

Ok, let me know if you want to take it forward, else I'll finish it now. I've been busy past few months.

csabella · 2020-05-23T15:44:14Z

@rotuna, please respond to @CuriousLearner's question and the review on your PR if you would like to finish this PR. My opinion is that @CuriousLearner should finish it on this PR if he's available to work on it now, but I'll let the two of you decide.

rotuna · 2020-05-27T19:59:11Z

Hi, sorry I didn't get an email about this and I've been caught up in other things. If @CuriousLearner wants to finish this that'd be perfectly fine with me. If he's not free I can try to finish it up next week.

* master: (1985 commits) bpo-40179: Fix translation of #elif in Argument Clinic (pythonGH-19364) bpo-35967: Skip test with `uname -p` on Android (pythonGH-19577) bpo-40257: Improve help for the typing module (pythonGH-19546) Fix two typos in multiprocessing (pythonGH-19571) bpo-40286: Use random.randbytes() in tests (pythonGH-19575) bpo-40286: Makes simpler the relation between randbytes() and getrandbits() (pythonGH-19574) bpo-39894: Route calls from pathlib.Path.samefile() to os.stat() via the path accessor (pythonGH-18836) bpo-39897: Remove needless `Path(self.parent)` call, which makes `is_mount()` misbehave in `Path` subclasses. (pythonGH-18839) bpo-40282: Allow random.getrandbits(0) (pythonGH-19539) bpo-40302: UTF-32 encoder SWAB4() macro use a|b rather than a+b (pythonGH-19572) bpo-40302: Replace PY_INT64_T with int64_t (pythonGH-19573) bpo-40286: Add randbytes() method to random.Random (pythonGH-19527) bpo-39901: Move `pathlib.Path.owner()` and `group()` implementations into the path accessor. (pythonGH-18844) bpo-40300: Allow empty logging.Formatter.default_msec_format. (pythonGH-19551) bpo-40302: Add pycore_byteswap.h header file (pythonGH-19552) bpo-40287: Fix SpooledTemporaryFile.seek() return value (pythonGH-19540) Minor modernization and readability improvement to the tokenizer example (pythonGH-19558) bpo-40294: Fix _asyncio when module is loaded/unloaded multiple times (pythonGH-19542) Fix parameter names in assertIn() docs (pythonGH-18829) bpo-39793: use the same domain on make_msgid tests (python#18698) ...

CuriousLearner · 2020-05-29T08:27:06Z

No problem. I'll just finish it up. Thanks for your work :)

* 'master' of github.com:python/cpython: (497 commits) bpo-40061: Fix a possible refleak in _asynciomodule.c (pythonGH-19748) bpo-40798: Generate a different message for already removed elements (pythonGH-20483) closes bpo-29017: Update the bindings for Qt information with PySide2 (pythonGH-20149) bpo-39885: Make IDLE context menu cut and copy work again (pythonGH-18951) bpo-29882: Add an efficient popcount method for integers (python#771) Further de-linting of zoneinfo module (python#20499) bpo-40780: Fix failure of _Py_dg_dtoa to remove trailing zeros (pythonGH-20435) Indicate that abs() method accept argument that implement __abs__(), just like call() method in the docs (pythonGH-20509) bpo-39040: Fix parsing of email mime headers with whitespace between encoded-words. (pythongh-17620) bpo-40784: Fix sqlite3 deterministic test (pythonGH-20448) bpo-30064: Properly skip unstable loop.sock_connect() racing test (pythonGH-20494) Note the output ordering of combinatoric functions (pythonGH-19732) bpo-40474: Updated coverage.yml to better report coverage stats (python#19851) bpo-40806: Clarify that itertools.product immediately consumes its inpt (pythonGH-20492) bpo-1294959: Try to clarify the meaning of platlibdir (pythonGH-20332) bpo-37878: PyThreadState_DeleteCurrent() was not removed (pythonGH-20489) bpo-40777: Initialize PyDateTime_IsoCalendarDateType.tp_base at run-time (pythonGH-20493) bpo-40755: Add missing multiset operations to Counter() (pythonGH-20339) bpo-25920: Remove socket.getaddrinfo() lock on macOS (pythonGH-20177) bpo-40275: Fix test.support.threading_helper (pythonGH-20488) ...

CuriousLearner · 2020-05-30T16:01:42Z

The Travis shows error here:

  File "/home/travis/build/python/cpython/Lib/test/test_imaplib.py", line 583, in <module>

    class ThreadedNetworkedTests(unittest.TestCase):

  File "/home/travis/build/python/cpython/Lib/test/test_imaplib.py", line 936, in ThreadedNetworkedTests

    @reap_threads

NameError: name 'reap_threads' is not defined

We haven't touch that portion at all and reap_threads is already imported in the file. Not sure what is the issue here. @vstinner

vstinner · 2020-06-01T18:03:21Z

You should use @threading_helper.reap_threads.

CuriousLearner · 2020-06-01T19:10:56Z

@vstinner Thanks, issue fixed. Can you please take a look now?

vstinner · 2020-06-02T01:17:57Z

Merged, thanks for the fix.

bpo-26543: Fix crash on IMAP4.noop when bytes data is passed.

5e698af

CuriousLearner requested a review from a team as a code owner August 11, 2019 11:08

the-knights-who-say-ni added the CLA signed label Aug 11, 2019

bedevere-bot added the awaiting review label Aug 11, 2019

CuriousLearner mentioned this pull request Aug 11, 2019

bpo-26543: fix imaplib.noop() failing when debug > 3 #1503

Closed

bpo-26543: Return the bad typ in case of noop

d43777f

maxking requested changes Oct 6, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Oct 6, 2019

vstinner reviewed Oct 7, 2019

View reviewed changes

rotuna mentioned this pull request Apr 23, 2020

bpo-26543: Fix crash on IMAP4.noop when bytes data is passed. #19693

Closed

CuriousLearner added 2 commits May 29, 2020 22:52

bpo-26543: Address review comments

cf11ead

remove handling NOOP in _command_complete method

f60e426

CuriousLearner changed the title ~~bpo-26543: Fix crash on IMAP4.noop when bytes data is passed.~~ bpo-26543: Fix IMAP4.noop when debug mode is enabled May 30, 2020

CuriousLearner added 2 commits May 30, 2020 17:21

fix name space for cpython_only decorator

acf6dab

csabella requested a review from vstinner May 31, 2020 11:44

fix import for reap_threads

7b20ca6

vstinner merged commit 8a3d2af into python:master Jun 2, 2020

bedevere-bot removed the awaiting changes label Jun 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-26543: Fix IMAP4.noop when debug mode is enabled #15206

bpo-26543: Fix IMAP4.noop when debug mode is enabled #15206

CuriousLearner commented Aug 11, 2019 •

edited by bedevere-bot

CuriousLearner commented Aug 11, 2019

maxking left a comment

maxking Oct 6, 2019

maxking Oct 6, 2019

vstinner Oct 7, 2019

bedevere-bot commented Oct 6, 2019

vstinner Oct 7, 2019

vstinner Oct 7, 2019

vstinner Oct 7, 2019

csabella commented Jan 12, 2020

rotuna commented Apr 23, 2020

CuriousLearner commented Apr 27, 2020

csabella commented May 23, 2020

rotuna commented May 27, 2020

CuriousLearner commented May 29, 2020

CuriousLearner commented May 30, 2020

vstinner commented Jun 1, 2020

CuriousLearner commented Jun 1, 2020

vstinner commented Jun 2, 2020

		@@ -0,0 +1 @@
		Fix crash on :meth:`IMAP4.noop()` when bytes data is passed.

bpo-26543: Fix IMAP4.noop when debug mode is enabled #15206

bpo-26543: Fix IMAP4.noop when debug mode is enabled #15206

Conversation

CuriousLearner commented Aug 11, 2019 • edited by bedevere-bot

CuriousLearner commented Aug 11, 2019

maxking left a comment

Choose a reason for hiding this comment

maxking Oct 6, 2019

Choose a reason for hiding this comment

maxking Oct 6, 2019

Choose a reason for hiding this comment

vstinner Oct 7, 2019

Choose a reason for hiding this comment

bedevere-bot commented Oct 6, 2019

vstinner Oct 7, 2019

Choose a reason for hiding this comment

vstinner Oct 7, 2019

Choose a reason for hiding this comment

vstinner Oct 7, 2019

Choose a reason for hiding this comment

csabella commented Jan 12, 2020

rotuna commented Apr 23, 2020

CuriousLearner commented Apr 27, 2020

csabella commented May 23, 2020

rotuna commented May 27, 2020

CuriousLearner commented May 29, 2020

CuriousLearner commented May 30, 2020

vstinner commented Jun 1, 2020

CuriousLearner commented Jun 1, 2020

vstinner commented Jun 2, 2020

CuriousLearner commented Aug 11, 2019 •

edited by bedevere-bot