Could PY3_BASE_MODULES be dynamic?

[byehack]

At #7836 by removing pathlib from PY3_BASE_MODULES we able to decrease the final size more than 30% (compressed: 3.6MB -> 2.3MB), can we still improve it?

IMO we could use dynamic list for base_library.zip modules:

1. normal modulefinder for bootstrap scripts.
2. include submodules of discovered modules (to avoid multiple search paths in runtime).
3. apply hooks on those scripts (e.g. excludeimports=["pathlib", "tokenize"] for pyimod02_importers.py).

[bwoodsend]

The only other package with submodules in base-library.zip is encodings where which submodules are needed is dependent on the runtime environment rather than the application (i.e. we can't strip it down)?

[byehack]

@rokm @bwoodsend What do you think?

[rokm]

1. Avoids problems like 6cb49d9 created in futures.

How so? Dynamic analysis of bootstrap modules will literally pull pathlib and tokenize back into base modules, so to solve that, you would need hooks that exclude them. I fail to see how that improves situation.

2. No need to manually change PY3_BASE_MODULES when dependencies of bootstrap scripts modified.

Like I said, bootstrap modules and scripts are not dynamic, so this is a non-issue.

Having to adjust the list every now and then due to internal changes in stdlib also does not warrant what IMHO is an overengineered solution, either.

3. hooks will help to prevent include modules in base_library.zip that doesn't need on bootstrap time (e.g. posixpath on Windows or vice verca).

Let's not drag modulegraph and hooks where they are not needed. For posixpath vs ntpath, if you really wanted to include just the relevant one (and I think at this point you are micro-optimizing well in the area of diminishing returns...), the ternary 'posixpath' if 'posix' in sys.builtin_module_names else 'ntpath', from your other PR seems like a better option.

[rokm]

@rokm @bwoodsend What do you think?

I fail to see the benefit, especially considering the additional complexity.

[byehack]

How so? Dynamic analysis of bootstrap modules will literally pull pathlib and tokenize back into base modules, so to solve that, you would need hooks that exclude them. I fail to see how that improves situation.

Yes that exactly I mean, add support hooks for bootstraps.

overengineered solution, either.

I don't think needs lots of work, all stuffs already implemented(?).

the ternary 'posixpath' if 'posix' in sys.builtin_module_names else 'ntpath', from your other PR seems like a better option.

Let constant variable remains constant. (another condition here.)

[bwoodsend]

Likewise, I'd be opposed this change. None of your proposed benefits really sound like benefits to me. It just sounds like unnecessary chaos to me.

No need to manually change PY3_BASE_MODULES when dependencies of bootstrap scripts modified.

We only have to touch that constant once a year and it's definitely one of the easiest parts of adding new Python version support.

[byehack]

I don't have time currently, but I like work on it in the future.
Notes for myself:

1. replace PY3_INTERPRETER_DEPS with PY3_BASE_MODULES
2. analyze bootstrap deps.
3. hooks for bootstrap scripts.
4. base_library.zip: PY3_INTERPRETER_DEPS + bootstrap deps (with submodules if available).

[byehack]

• Note for PY3_INTERPRETER_DEPS (ref: nuitka):

(.venv) E:\test>python -s -S -v -c ""               
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# C:\python37x86\lib\encodings\__pycache__\__init__.cpython-37.pyc matches C:\python37x86\lib\encodings\__init__.py
# code object from 'C:\\python37x86\\lib\\encodings\\__pycache__\\__init__.cpython-37.pyc'
# C:\python37x86\lib\__pycache__\codecs.cpython-37.pyc matches C:\python37x86\lib\codecs.py
# code object from 'C:\\python37x86\\lib\\__pycache__\\codecs.cpython-37.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x013D8F50>
# C:\python37x86\lib\encodings\__pycache__\aliases.cpython-37.pyc matches C:\python37x86\lib\encodings\aliases.py
# code object from 'C:\\python37x86\\lib\\encodings\\__pycache__\\aliases.cpython-37.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x013F11B0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x013D87F0>
# C:\python37x86\lib\encodings\__pycache__\utf_8.cpython-37.pyc matches C:\python37x86\lib\encodings\utf_8.py    
# code object from 'C:\\python37x86\\lib\\encodings\\__pycache__\\utf_8.cpython-37.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x013FD070>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# C:\python37x86\lib\encodings\__pycache__\latin_1.cpython-37.pyc matches C:\python37x86\lib\encodings\latin_1.py
# code object from 'C:\\python37x86\\lib\\encodings\\__pycache__\\latin_1.cpython-37.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x013FD2B0>
# C:\python37x86\lib\__pycache__\io.cpython-37.pyc matches C:\python37x86\lib\io.py
# code object from 'C:\\python37x86\\lib\\__pycache__\\io.cpython-37.pyc'
# C:\python37x86\lib\__pycache__\abc.cpython-37.pyc matches C:\python37x86\lib\abc.py
# code object from 'C:\\python37x86\\lib\\__pycache__\\abc.cpython-37.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x013FD4F0>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x013FD350>

As same as here, the only stdlibs need to be include are io, abc, encodings, codecs