a function to fetch a python file and automatically fetch (relative import) used modules, (optional) run them once

[grimmer0125]

From my perspective, there are two applications of Pyodide, one is REPL, the other one is to load Python script(s) to run and it could be a small piece of a code or a larger codebase. I have migrated my Chrome extension to Pyodide base from pure JS, and been published. Thank you for this great tool. I may want to refactor my codes to multiple files and add more functionalities. It would be great if there is such a feature/API introduced in the title. With it, I do not need to fetch one by one Python files and run them.

Supporting the overall Python native import system may be a difficult/long-term task, and starting from "relative import" or even limit to the same folder might be easier. I'm not sure whether this approach is good or not.

[ryanking13]

I agree. Though we can fetch external files then execute it, people will simply want to import module.

For that, I think we may extend import system such as:

• Fetch external python files,
• Save it to virtual file system,
• Then import it

The main problem of this approach will be how to properly fetch external python files that are import candidates
and verify that they are properly downloaded.

[hoodmane]

I'll try making it for a Pyodide package, since it will require js fetch (and readFileSync maybe for node).

It is perfectly possible to stick a Pyodide-only package on PyPi. Dunno if it is discouraged. I did that with pyodide-interrupts which is possible to import and use outside of Pyodide but also kind of useless. Though it's also useless to use in the currently version of Pyodide because we added the keyboard interrupt system.

But probably the best approach is indeed to include it as part of Pyodide.

we will need to provide documentation for this

The documentation is arguably the most important step.

[rth]

Maybe let's open separate issues to discuss points 2 and 3 and how the API should look before implementing it? The collapsible threads in "discussions" don't seem designed for long discussions.

Pyodide package with hacks Python import hook

Wouldn't it be easier to make a PyPi package for it, as a start?

pyodide.installFiles

Isn't this somewhat what the importmap system is about? We can certainly come up with our or own API, but it would need more design work to make sure it's right (and to avoid more confusion with existing installation methods). What I like about about installFiles is that it could be used on other files than Python ones though.

[rth]

It is perfectly possible to stick a Pyodide-only package on PyPi. Dunno if it is discouraged.

I think we should encourage putting packages to PyPi rather than vendoring them in Pyodide, when possible. Particularly for experimental stuff, it allows to move and iterate much faster (no need to wait for a Pyodide release for a bugfix). There are plenty of packages that don't make sense by themselves, all the pytest-* plugins, django-* plugins etc. From a Python package point of view all Pyodide does is to expose a few extra imports (e.g. js) which is fairly common. So as long as the name is pre-fixed with pyodide- I think it should be fine. We should also have a list of such packages somewhere.

[rth]

name is pre-fixed with pyodide-

BTW if you are interested @ryanking13 we could also host that future package (once it's ready) at the Pyodide Github org, so it's a bit easier to discover.

[ryanking13]

Maybe let's open separate issues to discuss points 2 and 3 and how the API should look before implementing it?

I made a proposal to discuss option 2. #1917

we could also host that future package (once it's ready) at the Pyodide Github org

Sure, I would very much appreciate that!

[grimmer0125]

Also worth reading the alternative solutions they have evaluated and in particular
...wasting bandwidth and time...

I think it might not be a big deal for small or medium size of a group of Python files. But it indeed needs to do some time measurement for any new feature for module resolution/hook/hack.

@ryanking13 Brython import system looks good.

--

Let me explain more about the initial proposal.

• I saw pyodide.loadPackagesFromImports and guess it already implement some kind of detect/scan the used import statement list in one python script string. So this is one of the reasons I propose to do a similar but more complicated thing.

• await pyodide.fetchAndRunPythonAsync(http://localhost/1.py) and it will automatically download the used user-defined and imported files, e.g. http://localhost/2.py used in 1.py.

  Originally what I imagine is a user defines from .module2 import sample_func in module1.py file, Pyodide can detect this and fetch module2 python file. There are two benefit

  1. It is straightforward and uses the same import syntax as the original Python module resolution.
  2. If the Python scripts do not use any Pyodide specific Python API inside, people can use local Python to do any tests for multiple Python files as they use the same import syntax. But I realize this case is not a common case when using Pyodide. So this benefit is not very much.

• Regarding wheel or different user scenarios about dev/deploy/production environments. I may not clarify my ideas in my previous comment very well. There are 3 cases

  1. For local development, the proposed feature (auto-detect/fetch) in this discussion is helpful, or alternatively, some auto-live-bundle wheel mechanisms may have the same benefit as people need to see the result from the code change sooner.
  2. To deploy a web project composed of JavaScript files and Pyodide scripts to some cloud, they often belong to the same GitHub repo (mono repo), personally, I think packaging Python scripts as a wheel may be acceptable. However, packaging as a wheel is still not straightforward for a self-own project, instead of shipping a library. Also, it needs some effort to set up it if using traditional setuptool. Poetry build almost does not pay any effort but it has some limitations on your project folder structure, Standard project structure not specified or not easily findable python-poetry/poetry#2252, which requires people to put codes in a first level folder and name the folder as the project name in pyproject.toml.
  3. To deploy a web project composed of JavaScript files which use some Python scripts belonging to another project for Pyodide usage, in this case, these Python files are not in the same cloud server, and packaging them as wheels seem very reasonable.

• We can put a warning like "really please only use this on localhost".

  @hoodmane,
  It seems that you have some concerns to apply this (implicit auto module resolution) other than local dev. It may be risky/unstable and encounter some http/resolution error so that it is hard to give the same guarantee as pyodide.installFiles design which uses an explicit file list instead of module resolution. Is this your concern? This is just a confirmation with you to avoid my incorrect understanding since I'm not involved in the above discussion. If so, I agree that this kind of feature may not become stable in a short time, using Python wheels is proven.

[grimmer0125]

hi, I notice Python package management Poetryhas solved the mentioned issue above, python-poetry/poetry#2252. Although it is not directly related to this topic, it may be still valuable to share.

in the generated pyproject.toml, you can customize the packages folder name and location, then they can differ to the default project/package , e.g. put your code inside a abcd folder, abc is the new package name.

[tool.poetry]
name = "pyodide-helper"

packages = [
    { include = "abcd", from = "." }
]

Then from abcd import test.

ref: https://github.com/python-poetry/poetry/pull/3257/files

This simplifies the building wheel setup.