stubgen doesn't find c modules in namespace of python package

[clark-ja]

Bug Report

stubgen does not find c extension modules that are placed in the namespace of a "regular" python package.

More Details:
For "regular" python packages and modules stubgen is using pkgutil.walk_packages() here to find subpackages/submodules but is not checking the namespace so it can miss some.

To Reproduce

This is how I encountered the issue (there is probably a simpler way to reproduce but I'm using gdb here as an example)

(Write your steps here:)

1. get gdb with python3 support (install or download a statically compiled version)
2. prepare a little wrapper script to start stubgen:

import sys
import mypy

mypy.util.check_python_version("stubgen")

options = mypy.stubgen.Options(
  pyversion=sys.version_info[:2],
  no_import=False,
  doc_dir="",
  search_path="",
  interpreter="",
  ignore_errors=False,
  include_private=False,
  output_dir="out",
  modules=[],
  packages=["gdb"],
  files=[],
  verbose=False,
  quiet=False,
  export_less=False
)

mypy.stubgen.generate_stubs(options)

3. start gdb and then source the previously written script: source /path/to/script.py

(these steps are necessary because the gdb python package is only accessible through the python interpreter inside GDB)

Expected Behavior

I was expecting the pyi files to contain all objects in the gdb namespace (including the c modules that gdb puts in the gdb namespace).

Actual Behavior

E.g. gdb.Architecture (and all the other c modules) are not included (can be quickly checked by running grep -r "Architecture" inside out dir) even though they are easily accessible from withing gdb: python help(gdb.Architecture).

Your Environment

I tried it in several environments before investigating, e.g.:

• Mypy version used: both mypy 0.761 and master (at commit 129dba4)
• Mypy configuration options from mypy.ini (and other config files): I used a clean env without additional configs
• Python version used: Python 3.8.10
• Operating system and version: Ubuntu 20.04.3 LTS x86_64