import of submodule polutes global namespace

[maxbachmann]

BPO	46935
Nosy	@sweeneyde, @maxbachmann

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-06.12:20:07.990>
created_at = <Date 2022-03-05.22:26:32.369>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10', 'invalid']
title = 'import of submodule polutes global namespace'
updated_at = <Date 2022-03-06.12:20:07.989>
user = 'https://github.com/maxbachmann'

bugs.python.org fields:

activity = <Date 2022-03-06.12:20:07.989>
actor = 'maxbachmann'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-06.12:20:07.990>
closer = 'maxbachmann'
components = ['Interpreter Core']
creation = <Date 2022-03-05.22:26:32.369>
creator = 'maxbachmann'
dependencies = []
files = []
hgrepos = []
issue_num = 46935
keywords = []
message_count = 4.0
messages = ['414599', '414600', '414603', '414617']
nosy_count = 2.0
nosy_names = ['Dennis Sweeney', 'maxbachmann']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46935'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[maxbachmann]

In my environment I installed the following two libraries:

pip install rapidfuzz
pip install python-Levenshtein

Those two libraries have the following structures:
rapidfuzz
|-distance
|- __init__.py (from . import Levenshtein)
|- Levenshtein.*.so
|-init.py (from rapidfuzz import distance)

Levenshtein
|-init.py

When importing Levenshtein first everything behaves as expected:

>>> import Levenshtein
>>> Levenshtein.
Levenshtein.apply_edit(       Levenshtein.jaro_winkler(     Levenshtein.ratio(
Levenshtein.distance(         Levenshtein.matching_blocks(  Levenshtein.seqratio(
Levenshtein.editops(          Levenshtein.median(           Levenshtein.setmedian(
Levenshtein.hamming(          Levenshtein.median_improve(   Levenshtein.setratio(
Levenshtein.inverse(          Levenshtein.opcodes(          Levenshtein.subtract_edit(
Levenshtein.jaro(             Levenshtein.quickmedian(   
>>> import rapidfuzz
>>> Levenshtein.
Levenshtein.apply_edit(       Levenshtein.jaro_winkler(     Levenshtein.ratio(
Levenshtein.distance(         Levenshtein.matching_blocks(  Levenshtein.seqratio(
Levenshtein.editops(          Levenshtein.median(           Levenshtein.setmedian(
Levenshtein.hamming(          Levenshtein.median_improve(   Levenshtein.setratio(
Levenshtein.inverse(          Levenshtein.opcodes(          Levenshtein.subtract_edit(
Levenshtein.jaro(             Levenshtein.quickmedian( 

However when importing rapidfuzz first it import rapidfuzz.distance.Levenshtein when running import Levenshtein

>>> import rapidfuzz
>>> Levenshtein
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'Levenshtein' is not defined
>>> import Levenshtein
>>> Levenshtein.
Levenshtein.array(                  Levenshtein.normalized_distance(    Levenshtein.similarity(
Levenshtein.distance(               Levenshtein.normalized_similarity(  
Levenshtein.editops(                Levenshtein.opcodes( 

My expectation was that in both cases import Levenshtein should import the Levenshtein module. I could reproduce this behavior on all Python versions I had available (Python3.8 - Python3.10) on Ubuntu and Fedora.

[maxbachmann]

It appears this only occurs when a C Extension is involved. When the so is imported first it is preferred over the .py file that the user would like to import. I could not find any documentation on this behavior, so I assume that this is not the intended.

My current workaround is the usage of a unique name for the C Extension and the importing everything from a Python file with the corresponding name.

[sweeneyde]

This might be something that rapidfuzz can fix, rather than CPython. In whatever import process rapidfuzz uses, it populates sys.modules with a module named Levenshtein in addition to 'rapidfuzz.distance.Levenshtein'. You might be able to request that they change something there.

Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> m1 = set(sys.modules.keys())
>>> import numpy
>>> m2 = set(sys.modules.keys())
>>> import rapidfuzz
>>> m3 = set(sys.modules.keys())
>>> m2 - m1
{'posixpath', 'numpy.random._pcg64', 'numpy.fft._pocketfft_internal', 'numpy.core._dtype', 'cython_runtime', 'numpy.random._sfc64', 'urllib', 'numpy.core.einsumfunc', 'sre_parse', 'numpy.polynomial.laguerre', 'numpy.fft._pocketfft', 'numpy.lib._iotools', 'numpy.core._exceptions', 'numpy.random._pickle', 'numpy.lib.twodim_base', 'numpy.polynomial._polybase', 'numpy.lib._datasource', 'copyreg', 'numpy.random._philox', '_ctypes', 'numpy.core._ufunc_config', 'platform', 'numpy.lib.histograms', 'numpy.lib.mixins', 'numpy.core._asarray', 'numpy.polynomial.legendre', 'numpy.polynomial.polyutils', 'subprocess', 'fnmatch', 'numpy.ctypeslib', 'ast', 'sre_compile', 'json.scanner', 'numpy.core.arrayprint', 'textwrap', 'numpy.core.multiarray', 'datetime', 'inspect', 'numpy.core.function_base', 'hmac', 'numpy.lib.utils', '_json', 'signal', 'numpy.core.machar', 'numpy.ma.core', 'pathlib', 'numbers', 'numpy.core._methods', 'numpy.lib.type_check', 'numpy.core.defchararray', 'numpy.core.getlimits', 'numpy.lib.ufunclike', 'numpy.version', 'select', '_sre', 'numpy.core._dtype_ctypes', 'numpy.lib.arrayterator', 'random', '_blake2', 'numpy.fft', 'token', 'numpy.core._string_helpers', 'numpy', '_hashlib', 'opcode', 'tokenize', 'numpy.random._bounded_integers', 'numpy.random.mtrand', 'ctypes._endian', '_weakrefset', 'numpy.ma', 'numpy.lib.nanfunctions', '_random', 'numpy.lib.function_base', '_sha512', 'bisect', 'numpy.core.records', 'numpy._globals', '_compat_pickle', 'urllib.parse', 'numpy.random.bit_generator', 'numpy.linalg._umath_linalg', 'numpy.core._add_newdocs_scalars', 'numpy.polynomial.hermite', 'base64', 'numpy.linalg.linalg', 'numpy.core._multiarray_tests', '_cython_0_29_24', 'hashlib', '_struct', 'numpy.lib.arraypad', 'numpy.core', 'msvcrt', 'numpy.ma.extras', 'numpy.lib.index_tricks', '_locale', 'numpy.lib.shape_base', 'numpy.compat._inspect', 'numpy.polynomial.hermite_e', 'pickle', 'numpy._distributor_init', 'numpy.lib._version', '_datetime', 'secrets', 'numpy.lib.polynomial', 'numpy.core.numerictypes', '_ast', 'numpy.lib.scimath', '_winapi', 'numpy.matrixlib.defmatrix', '_socket', 'numpy.core.shape_base', 'numpy.lib.format', 'dis', 'numpy.core._multiarray_umath', 'weakref', 'numpy.compat.py3k', 'json', 'numpy.core.umath', 'numpy.core.numeric', 'numpy.core.memmap', 'sre_constants', 'numpy.compat', 'numpy.core._add_newdocs', 'numpy.polynomial.chebyshev', 'math', 'numpy.random._common', 'numpy.linalg', 'numpy.random', 're', 'threading', 'numpy._pytesttester', '_bisect', 'collections.abc', 'socket', 'numpy.lib.stride_tricks', 'linecache', 'numpy.lib', 'numpy.fft.helper', 'numpy.core.fromnumeric', 'json.encoder', 'numpy.linalg.lapack_lite', 'selectors', 'numpy.polynomial', 'numpy.core._internal', 'numpy.__config__', 'numpy.polynomial.polynomial', 'numpy._version', 'errno', 'struct', 'ctypes', 'numpy.random._mt19937', 'binascii', 'numpy.lib.npyio', 'numpy.random._generator', 'numpy.lib.arraysetops', 'numpy.matrixlib', '_opcode', 'json.decoder', 'numpy.core._type_aliases', 'enum', 'numpy.core.overrides', '_pickle'}
>>> m3 - m2
{'rapidfuzz.process', 'rapidfuzz.cpp_process', 'rapidfuzz.string_metric', 'rapidfuzz.distance.Levenshtein', 'rapidfuzz.cpp_utils', 'rapidfuzz.cpp_process_cdist', 'cpp_utils', 'cpp_process_cdist', 'rapidfuzz', 'Jaro', '_heapq', 'rapidfuzz.distance.JaroWinkler', 'cpp_fuzz', 'rapidfuzz.distance.Hamming', '_cython_3_0_0a10', 'rapidfuzz.utils', 'cpp_process', 'Hamming', 'rapidfuzz.distance', 'rapidfuzz.distance.Jaro', 'JaroWinkler', '_initialize', 'rapidfuzz.cpp_fuzz', 'cpp_string_metric', 'rapidfuzz.distance._initialize', 'rapidfuzz.distance.Indel', 'rapidfuzz.fuzz', 'Levenshtein', 'heapq', 'Indel', 'array', 'rapidfuzz.cpp_string_metric'}
>>> sys.modules['rapidfuzz.distance.Levenshtein'] is sys.modules['Levenshtein']
True

A temporary workaround could be to delete sys.modules['Levenshtein']:

>>> import rapidfuzz
>>> del sys.modules['Levenshtein']
>>> import Levenshtein
>>> Levenshtein.__file__ # not the rapidfuzz one
'C:\\Users\\sween\\AppData\\Roaming\\Python\\Python310\\site-packages\\Levenshtein\\__init__.py'

[maxbachmann]

Thanks Dennis. This helped me track down the issue in rapidfuzz.