Allow to restrict ModuleFinder to get "direct" dependencies

[miscfrommetz]

BPO	1284670
Nosy	@theller, @ncoghlan, @merwok, @ethanfurman, @ericsnowcurrently, @JelleZijlstra, @iritkatriel, @MaxwellDupre
PRs	bpo-1284670: Allow to restrict ModuleFinder to get direct dependencies  #30228
Files	modulefinder.no_recurse.diff: Add a parameter to ModuleFinder, to not recurse when searching module. modulefinder-no-recurse.diff 1.html: bpo-1284670 2.html: fix non_recursive_import_test_1 and set debug to 0

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/merwok'
closed_at = None
created_at = <Date 2005-09-08.09:28:31.000>
labels = ['easy', 'type-feature', 'library']
title = 'Allow to restrict ModuleFinder to get "direct" dependencies'
updated_at = <Date 2022-01-20.17:22:41.162>
user = 'https://bugs.python.org/miscfrommetz'

bugs.python.org fields:

activity = <Date 2022-01-20.17:22:41.162>
actor = 'JelleZijlstra'
assignee = 'eric.araujo'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2005-09-08.09:28:31.000>
creator = 'misc_from_metz'
dependencies = []
files = ['6778', '22792', '50466', '50467']
hgrepos = []
issue_num = 1284670
keywords = ['patch', 'easy']
message_count = 16.0
messages = ['48725', '48726', '48727', '114551', '114553', '114645', '114646', '124928', '140602', '141381', '406579', '406981', '407139', '407470', '407471', '411038']
nosy_count = 11.0
nosy_names = ['nnorwitz', 'theller', 'jvr', 'ncoghlan', 'eric.araujo', 'misc', 'ethan.furman', 'eric.snow', 'JelleZijlstra', 'iritkatriel', 'mikecmcleod']
pr_nums = ['30228']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1284670'
versions = ['Python 3.3']

[miscfrommetz]

this patch add a parameter to ModuleFinder, to not
recurse when searching module.
This way, someone can compute the modules needed by a
specific file, without having to compute all other
imported module, which may take time for result that
are no t needed.

With the current system, you cannot create a clean
graph of modules dependancy, since they are all on the
same level.

This patch do not change the default behavior.

[nnorwitz]

Logged In: YES
user_id=33168

Thomas are you the defacto maintainer of modulefinder? Any
comments?

[theller]

I have no time or interest in this patch, so unassigning.

[BreamoreBoy]

Anyone interested in this or can it be closed?

[merwok]

Adding jvr and people interested in import machinery, per Misc/maintainers.rst

Michael, can you refresh your patch against the py3k branch? Please also generate a unified diff from the top level of the source distribution, as detailed in http://www.python.org/dev/patches/ The feature seems useful to me and the patch is straightforward. Tests and doc updates are needed; do you want to add them?

Mark, I think that feature requests are never closed just because they’re old; they have to be rejected, implemented or obsoleted. (Thank you for all your triage work.)

[ncoghlan]

The generalist in me is inclined to suggest a "depth" parameter (with depth=1 equivalent to direct dependencies only, and depth = None meaning all dependencies), but I must admit I don't have a concrete use case for the extra generality. So the simpler, recurse/don't recurse approach is probably a better option (building a depth-limited search on top of the recursion flag wouldn't be difficult anyway).

Aside from missing docs and unit test updates, the idea seems sound.

[ncoghlan]

Although I do find it a little concerning that there is no mention of sys.path_hooks or sys.meta_path in the modulefinder source code. I suspect this module only works correctly with vanilla filesystem based imports and can't handle anything imported via PEP-302.

[merwok]

The depth parameter idea sounds like YAGNI, so let’s stay with a recurse boolean :)

[merwok]

I applied the patch, added a test and found a bug. Here’s my progress so far; someone can start from it to write more tests and fix the code.

[merwok]

I modernized modulefinder a bit in 1521d9837d16; here’s a refreshed patch.

[MaxwellDupre]

I would like to help with this issue.

I note the history (using Git Dag) of the modulefinder.py shows that it has been modified with changes related here:

Author: Éric Araujo <merwok@netwok.org> 2011-07-28 22:35:29
Committer: Éric Araujo <merwok@netwok.org> 2011-07-28 22:35:29
Parent: 867754e (merge 11164)
Child: cee6bb5 (Fix bug I unwittingly added in 1521d9837d16 (found by Ezio Melotti))
Branches: main, remotes/origin/3.10, remotes/origin/3.6, remotes/origin/3.7, remotes/origin/3.8, remotes/origin/3.9, remotes/origin/bpo-45514/deprecate-importlib-resources-legacy, remotes/origin/buildbot-custom, remotes/origin/main, remotes/upstream/3.10, remotes/upstream/3.6, remotes/upstream/3.7, remotes/upstream/3.8, remotes/upstream/3.9, remotes/upstream/bpo-45514/deprecate-importlib-resources-legacy, remotes/upstream/buildbot-custom, remotes/upstream/main, remotes/upstream/windeepfreeze
Follows: v3.2.1
Precedes: v3.3.0a1

Modernize modulefinder module and tests a bit.

The tests don’t use an internal distutils function anymore, and use
regular assertEqual with sorted lists instead of a convoluted manual
diff.

and

Author: Éric Araujo <merwok@netwok.org> 2011-08-01 14:29:07
Committer: Éric Araujo <merwok@netwok.org> 2011-08-01 14:29:07
Parent: 1e3a68d (Modernize modulefinder module and tests a bit.)
Child: cb66eb0 (Issue bpo-13959: Deprecate imp.get_suffixes() for new attributes on)
Branches: main, remotes/origin/3.10, remotes/origin/3.6, remotes/origin/3.7, remotes/origin/3.8, remotes/origin/3.9, remotes/origin/bpo-45514/deprecate-importlib-resources-legacy, remotes/origin/buildbot-custom, remotes/origin/main, remotes/upstream/3.10, remotes/upstream/3.6, remotes/upstream/3.7, remotes/upstream/3.8, remotes/upstream/3.9, remotes/upstream/bpo-45514/deprecate-importlib-resources-legacy, remotes/upstream/buildbot-custom, remotes/upstream/main, remotes/upstream/windeepfreeze
Follows: v3.2.1
Precedes: v3.3.0a1

Fix bug I unwittingly added in 1521d9837d16 (found by Ezio Melotti)

And the modulefinder.py file has been subsequently modified by another 28 changes.

So I am wondering if this issue is still relevant?

[iritkatriel]

Mike, from looking at the code the change proposed here is not there, so while the patch may not apply cleanly anymore, the commits you mention do not make this issue irrelevant.

[MaxwellDupre]

Ok, I will work on this soon and make further comments.

[MaxwellDupre]

I have made the changes as indicated in the diff files. I have tested against the latest from my GitHub copy.
The result is:
./python -m test -j0
== CPython 3.11.0a2+ (heads/main:3a91617590, Dec 1 2021, 15:11:41) [GCC 11.2.0]
== Linux-5.13.0-22-generic-x86_64-with-glibc2.34 little-endian
== cwd: /home/dougal/Documents/cpython/build/test_python_300237æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
...
0:01:34 load avg: 8.77 [221/430/1] test_modulefinder passed
== Tests result: FAILURE ==
409 tests OK.
1 test failed:
test_embed
20 tests skipped:
test_curses test_devpoll test_epoll test_ioctl test_kqueue
test_msilib test_ossaudiodev test_smtpnet test_socketserver
test_startfile test_tix test_tk test_ttk_guionly test_urllib2net
test_urllibnet test_winconsoleio test_winreg test_winsound
test_xmlrpc_net test_zipfile64
Total duration: 5 min 30 sec
Tests result: FAILURE

The first diff is attached. But there was a bug in the code and I will upload in next comment.

[MaxwellDupre]

Minor fix after test.

[JelleZijlstra]

Mike's fix unfortunately didn't work out.

What are the rules about closing old enhancement requests? modulefinder is an old and rarely used package, and I feel like the use case is better served by a PyPI package.