gh-93696: Locate frozen module source with __file__

[SnoopJ]

See #93696, this patch allows pdb to locate the source for frozen stdlib modules. With the patch applied, the output of pdb is as expected:

$ git describe && ./python -V
v3.11.0b1-358-gc91afc1c0a
Python 3.12.0a0
$ ./python -m pdb repro.py 
> /home/snoopjedi/repos/cpython/repro.py(2)<module>()
-> import importlib._bootstrap
(Pdb) n
> /home/snoopjedi/repos/cpython/repro.py(5)<module>()
-> importlib._bootstrap._resolve_name("os", ".", 1)
(Pdb) s
--Call--
> <frozen importlib._bootstrap>(1037)_resolve_name()
(Pdb) l
1032        def __exit__(self, exc_type, exc_value, exc_traceback):
1033            """Release the import lock regardless of any raised exceptions."""
1034            _imp.release_lock()
1035 
1036 
1037 -> def _resolve_name(name, package, level):
1038        """Resolve a relative module name to an absolute one."""
1039        bits = package.rsplit('.', level - 1)
1040        if len(bits) < level:
1041            raise ImportError('attempted relative import beyond top-level package')
1042        base = bits[0]

I am not particularly familiar with the frozen module system so I'm not sure if there are edge cases where this will produce undesirable behavior, but thought that I would send it upstream just in case 😅

• Issue: pdb cannot find source code for frozen stdlib modules #93696

[SnoopJ]

Edit: I have now added a test to this PR, see follow-up comment

I did not include an additional test with this PR because I don't think there's a good way to generate a dummy frozen module for use in the test and it seems inappropriate to test against one of the stdlib modules. All the same, here's what the test would look like for the current version of importlib._bootstrap, in case there's some clever testing trick I missed out here.

click to see patch

diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 0141b739c2..894f3e9099 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -532,6 +532,40 @@ def test_list_commands():
     (Pdb) continue
     """
 
+def test_frozen_list():
+    '''Test the list command on frozen stdlib modules
+
+    >>> def test_function():
+    ...     import importlib._bootstrap
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     importlib._bootstrap._resolve_name("os", ".", 1)
+
+    >>> with PdbTestInput([  # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
+    ...     'step',
+    ...     'list',
+    ...     'continue',
+    ... ]):
+    ...     test_function()
+    > <doctest test.test_pdb.test_frozen_list[0]>(4)test_function()
+    -> importlib._bootstrap._resolve_name("os", ".", 1)
+    (Pdb) step
+    --Call--
+    > <frozen importlib._bootstrap>(1037)_resolve_name()
+    (Pdb) list
+    1032            def __exit__(self, exc_type, exc_value, exc_traceback):
+    1033                """Release the import lock regardless of any raised exceptions."""
+    1034                _imp.release_lock()
+    1035
+    1036
+    1037 ->     def _resolve_name(name, package, level):
+    1038            """Resolve a relative module name to an absolute one."""
+    1039            bits = package.rsplit('.', level - 1)
+    1040            if len(bits) < level:
+    1041                raise ImportError('attempted relative import beyond top-level package')
+    1042            base = bits[0]
+    (Pdb) continue
+    '''
+
 def test_pdb_whatis_command():
     """Test the whatis command

[SnoopJ]

Updated with a test that creates a module that suitably fakes the behavior of functions inside of frozen modules, in the sense that co_filename on the contained function starts with '<frozen', and the __file__ attribute of the module will point to the "real" source file that pdb should use for resolving list

[cpython-cla-bot]

All commit authors signed the Contributor License Agreement.

[kumaraditya303]

LGTM

[SnoopJ]

@ericsnowcurrently does this patch need any changes?

[SnoopJ]

@JelleZijlstra generously offered to have a look-see at this during the 3.11 release stream, dropping this here to ping him :)

[JelleZijlstra]

A little weird that we're writing into the CWD, but that appears to be what the other tests in this file are doing too.

[JelleZijlstra]

What if __file__ is not a string? It looks like get_file_breaks will crash in that case. We should probably just ignore __file__ in that case.

[SnoopJ]

Hmm, that's a good point. I would think it's a small edge, but it's easy to be more cautious about it.

[JelleZijlstra]

Yes, I think it's important for a debugger to be resilient in the face of unusual program state.

[miss-islington]

Thanks @SnoopJ for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

[bedevere-bot]

GH-98653 is a backport of this pull request to the 3.11 branch.

[bedevere-bot]

GH-98654 is a backport of this pull request to the 3.10 branch.