Improved modulefinder

[theller]

BPO	462936
Nosy	@loewis, @theller
Files	mf.diff: Patch against current CVS version

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2002-04-12.17:13:41.000>
created_at = <Date 2001-09-19.17:43:03.000>
labels = ['extension-modules']
title = 'Improved modulefinder'
updated_at = <Date 2002-04-12.17:13:41.000>
user = 'https://github.com/theller'

bugs.python.org fields:

activity = <Date 2002-04-12.17:13:41.000>
actor = 'theller'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2001-09-19.17:43:03.000>
creator = 'theller'
dependencies = []
files = ['3641']
hgrepos = []
issue_num = 462936
keywords = ['patch']
message_count = 5.0
messages = ['37690', '37691', '37692', '37693', '37694']
nosy_count = 2.0
nosy_names = ['loewis', 'theller']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue462936'
versions = []

[theller]

This patch adds two improvements to
freeze/modulefinder.

1. ModuleFinder now keeps track of which module is
   imported by whom.

2. ModuleFinder, when instantiated with the new
   scan_extdeps=1 argument, tries to track dependencies
   of builtin and extension modules.

[loewis]

Logged In: YES
user_id=21627

I dislike the chunk on finding external dependencies. What
is the typical use case (i.e. what module has what
external dependencies)?

It seems easier to hard-code knowledge about external
dependencies into ModuleFinder; this hard-coded knowledge
should cover all core modules. In addition, there should
be an API to extend this knowledge for non-standard
modules.

Furthermore, by executing an import, you cannot be sure
that you really find all dependencies - some may only show
up when a certain function is used.

[theller]

Logged In: YES
user_id=11105

The use case is to find as much dependencies as possible.

Sure, you cannot assume that importing an extension module
finds all dependencies - only those which are executed
inside the initmodule function. OTOH, this covers a *lot*
of problematic cases, pygame and numpy for example.

The situation is (somewhat) similar to finding dependencies
of python modules - only those donewith normal import
statements are found, __import__, eval, or exec is not
handled.

A possible solution would be to run the script
in 'profiling mode', where the script is actually run, and
all imports are monitored. This is however far beyond
ModuleFinder's scope.

Hardcoding the knowledge about dependencies into
ModuleFinder for the core modules would be possible
although inelegant IMO. An API for non-standard modules
would be possible, but how should this be used without
executing any code?

[loewis]

Logged In: YES
user_id=21627

I would still prefer the inelegant approach. Your approach
appears to be quite dangerous: the packager would
essentially run arbitrary C code... If you absolutely have
to use such a feature, I think you can do better than
analysing the python -v output: watch sys.modules before
and after the import.

As for extending the hard-coded knowledge: I was
suggesting that the packaging tool that uses modulefinder
has a mechanism to extend the hard-coded knowledge by
other hard-coded knowledge (which lives in the packaging
tool, instead of living in modulefinder). If the packaging
tool absolutely wants to, it also could run the Python
interpreter through a pipe and put the gathered output
into modulefinder :-)

[theller]

Logged In: YES
user_id=11105

Closed as rejected.