Source-less python distributions + PEP3147

[vfazio]

PEP 3147 (https://peps.pythondiscord.com/pep-3147/) makes mention of "continue to support source-less distributions" by using legacy pyc file placements when no py file is adjacent. Blurb here: https://peps.pythondiscord.com/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports

What is not clear is if the term "legacy" applies to just the pyc file placement or if it applies to both the pyc file placement and source-less distributions.

Are source-less distributions first-class citizens in terms of support? They're often used for embedded environments (Buildroot, for example, generates a source-less distribution) so it would make sense that they are supported.

If source-less distributions are first-class citizens and should continue to be supported, is there documentation about what that implies for consumers of a distribution? Some things that we've seen are:

• Packages trying to symlink to .py files vs building a wrapper script that imports a module to execute code
• Modules making assumptions that a .py file which may-or-may-not exist can be executed directly

[tpetazzoni]

Hello @vstinner, do you think you could have a look at this issue that is affecting us in the Buildroot project? Thanks a lot!

[vfazio]

@warsaw As the author of the PEP, do you have thoughts on this?

[warsaw]

@vfazio I'm not sure exactly what you're asking. PEP 3147, Case 4 specifically describes pyc locations. I would say that if a pyc location is found outside of __pycache__ and there's no py file next to it, Python should continue to import the pyc. So by implication Python itself supports source-less imports. But that doesn't mean any 3rd party packages or other tools support a source-less mode of operation. I don't believe there is much if any documentation about it.

[vfazio]

@warsaw thanks for taking the time to respond.

I think the first and most important question is: does Python support and plan to continue to support source-less distributions?

If it's not a long term goal and will (or is even currently) a deprecated idea, then the rest is moot.

You mention "So by implication Python itself supports source-less imports", and I agree, this support appears to be only implied and not explicitly stated. If I read between the lines of PEP 3147, Case 4 it implies that source-less distributions are "legacy" because they only work with legacy file placements and likely not considered "supported", or "guaranteed", long term since there is no analogue for this distribution type with the "new" format in __pycache__.

If source-less distributions are not part of regression tests and are not taken into consideration when making future roadmap decisions then I would consider them unsupported and "try at your own risk".

If source-less distros are supported:

I think you answered the other question, that there is no documentation about considerations for source-less distros that packages need to take into account.

Having a clearly documented mention of support would be useful for consumers of the source-less distro to point to when raising issues to 3rd party packages. Short of some official documentation, at least being able to point to, say, this issue would be useful in saying "This method of distribution is intentional and supported, please consider adding support in your package".

This way we could go to PIP and point to https://github.com/vfazio/pip/blob/main/src/pip/_vendor/pep517/in_process/__init__.py#L13 and say "hey, that may not work for source-less distros"

If source-less distros are not supported:

As I mentioned in my previous comment, source-less distros are useful in embedded/space-constrained environments so having this ability is very useful and I'd ask that they be reconsidered.

@tpetazzoni @yann-morin-1998 please chime in with any other concerns you may have as it relates to BR

[warsaw]

If source-less distributions are not part of regression tests and are not taken into consideration when making future roadmap decisions then I would consider them unsupported and "try at your own risk".

Sourceless imports are tested.

Having a clearly documented mention of support would be useful for consumers of the source-less distro to point to when raising issues to 3rd party packages.

PEP 3147, __pycache__, and sourceless imports could be better documented. There's some discussion of __pycache__ in the importlib documentation but it doesn't talk about sourceless at all. Probably, there should be a section discussing this in the import system documentation, perhaps in the Path Based Finder section (although I forget the details about where exactly in importlib this is implemented).

Contributions welcome! Also @brettcannon for visibility.

[vfazio]

Hmm ok, i'll have to look into this further. I guess I was expecting to see more rigor than that a single sourceless import worked and more like the entire python3 test suite being run on a source-less distro. That's probably a github action we could add to ensure everything works as expected. We do have some limited evidence that sourceless distros work as i would have expected more issues would have cropped up from the Buildroot community if it didn't, but this would still be a good test to add.

I'm up for making a contribution if we can determine the correct place to add the documentation and discuss what the content should encompass including any known gotchas and exceptions.

[warsaw]

Sounds good. I will try to find some time to review your proposals / PRs.

[brettcannon]

I don't expect .pyc-only imports going anywhere, but we historically discourage them unless you have a pressing use-case because they make tracebacks way less useful. I actually tried to remove support for .pyc-only imports, but I got enough pushback that it didn't happen.

Also understand that .pyc-only imports are CPython-specific, so they can't be put into the language reference. I'm not sure where you would want any official documentation on this to be? There is https://docs.python.org/3/library/importlib.html#importlib.machinery.SourcelessFileLoader and so the mechanism is documented.

[yann-morin-1998]

we historically discourage them unless you have a pressing use-case

This is a situation that is very common on embedded devices:

• the size of the storage is relatively small, orders of magnitudes smaller than desktop and servers;
• the filesystem is very often read-only and also compressed (by design, like squashfs), it is impossible to write the pre-compiled modules in a PEP3147 compliant location;
• they are usualy lower-powered than traditional desktop or servers, so loading modules from source can be quite a big chunk of the startup of a python program.

So, it is very convenient to pre-compile the modules at build time, and drop the sources on production builds: this regains some space, and noticeably lowers startup time.