Some import tools for developing and debugging Brython apps/libraries

[BrenBarn]

Developing a Brython app is a somewhat painful process if you want to reuse existing pure-Python code as part of your app, and especially if you want to reuse it for multiple apps. Related issues have been raised in #491 and #1381. For me, a large part of the problem comes from the limitations that the browser imposes on the import process. I'm going to describe the kind of problems I encounter and then give an outline of some small features Brython could grow to help make this process a bit easier. Sorry for the length but I get the impression from other issues that other people are facing similar pain points so I want to explain where this is coming from and see if others agree.

For the TL;DR: I'm suggesting that the brython CLI include a simple dev server that maps requests for modules to the "normal" location of those module files in the actual site-packages directory of the installed Python.

Suppose I have a Python library somelib installed on my computer. It's a pure Python library, so it's usable by Brython. . . as long as I can get it in the right place. Typically I'll be working on my Brython app in some directory, say myapp. But somelib is installed as a normal Python lib in the site-packages directory.

Of course, I can't really use imports at all unless I run a webserver. So I run a simple test server to allow me to import local files. I'd like to set it up to serve the myapp directory, since that's where my app is. But then it doesn't have access to "normal" installed Python libs like somelib. Now, Brython has a solution of sorts for this: the --add-package command in the CLI, which will copy somelib into my app directory for me so that Brython can see it.

The problem is that this detaches that Brython-local copy of somelib from the installed Python version. If I update somelib, the Brython version is not updated.

This is especially painful when somelib is actually a library written with Brython in mind (let's call it brylib). I often find that, in writing an app, I want to factor out some code into a Brython-aware library so that I can reuse it in other apps. This means I'm often developing an individual app in parallel with a library (brylib) that accumulates "useful things" that I anticipate using in other apps (or that I've already used in other apps and want to put into a library instead of copy-pasting into my new app).

This process of parallel development is extremely unwieldy. The only way Brython can see brylib is if I put it in a subdirectory of myapp --- but this defeats the purpose of factoring the code out, because it's now "trapped" inside the directory of a particular app. I can put brylib somewhere else to develop it, but then I have to repeatedly run python -m brython --add_package brylib to update it every time I make a change. It's even worse if I'm developing multiple apps since I have to remember to do that separately in every app. Importantly, this isn't just something I have to do to prepare my app for release; I have to do it over and over every time I want to use an updated version of a library.

How can this be fixed? What I'd like to propose is that Brython include a simple development server, based on the http.server in the Python stdlib, which maps requests for Python libraries directly to Python source files located in the usual site-packages directory (i.e., outside the directory where the server is running).

Right now to resolve a statement like import somelib, Brython executes Ajax requests for somelib.py in the current directory, then somelib/__init__.py, and eventually Lib/site-packages/somelib.py and then Lib/site-packages/somelib/__init__.py and so on. The idea is that this development server would recognize these site-packages requests and return the contents of Python files located in the actual system site-packages directory. In other words, although you'd run the dev server in the /projects/myapp directory, it would actually be serving files from the "real" site-packages directory somewhere else on your computer. Under normal circumstances this might be considered a security risk, but for a dev server it's fine. (Actually I think even better would be a slight modification to this, which I'll mention in a bit.)

This would mean that, as long as I can fire up a regular Python session and do import somelib and it works, then it will also work in a Brython app running on the Brython dev server, because the dev server will just look in the same place that Python looks to import the same library. This means that, during the development process, your Brython code will have full access to any libraries on the local system (assuming, of course, that they're pure Python so Brython can translate them to JS).

Of course, when you want to release your app, you'll still have to do the --add_package/--modules dance, but you'll only have to do that as part of the release process, to "freeze" whatever state the libraries are in. You won't have to do it again and again during development to refresh the libraries.

I've managed to write a simple proof-of-concept server that does this. It basically looks like this:

prefix = '/Lib/site-packages/'
class ModuleFindingHandler(http.server.SimpleHTTPRequestHandler):
	def do_GET(self):
		# get rid of query string
		path = urllib.parse.urlparse(self.path).path
		if path.startswith(prefix):
			f = self.find_module(path)
			if f:
				try:
					self.copyfile(f, self.wfile)
				finally:
					f.close()
		else:
			super().do_GET()

I've left out the details of find_module there, but basically it uses site.getsitepackages() to find the site-packages directory and then tries to open a file there matching the one in the request.

The idea is that the Brython cli could grow a new option --dev-server that would run a server like this. In this way the process of developing Brython apps and libraries would be streamlined.

In fact, as I mentioned above, I think a slight variation on this would be even better. That is that the JS end of Brython could have a "dev mode" for its import system. In this mode, in addition to trying the usual requests to find a library, it would also try something like /get_local_library?libname=somelib. In other words, rather than asking for a specific filename, it would just pass the name of the library it's trying to import. This would allow the dev server to use the ordinary Python import mechanism (via importlib) to find the module, not just in site-packages, but anywhere that Python might normally find it (for instance in a user-specific package directory, or a directory added to the path with a .PTH file, or any such thing). This would require some cooperation from Brython to add this additional type of request to its import system. It could be specified in the HTML with an additional flag the way debug levels and such are, e.g., brython({'dev_import': true}).

Sorry for the long post but hopefully that makes clear the problem I'm trying to solve here. Basically I find that an important part of fluid Python development is the ability to "just import stuff", and the limitations of the browser prevent that. But during development, we can suppose that we don't just have the browser, we have a dev server, and that dev server can smooth things over to give Brython access to existing Python modules without having to actually copy them into the app directory until it's actually time for release.

Does this seem like something that would be useful?

[schmave]

Seems like a useful feature. Could some of this functionality could be achieved (on Mac/Linux at least) by make the Brython-local somelib folder be a symbolic link to the somelib files that you make changes to outside of Brython?

[BrenBarn]

It could, but that still requires there to be a Brython-local somelib folder in the first place. If the dev server looks directly in site-packages, it will just find any library it needs and you don't need to create a Brython-local folder for any libs until you're ready to release/deploy.

[PierreQuentel]

@BrenBarn Thanks for the proposal and the detailed explanation.

In the commits referenced above I have implemented a possible solution to this problem:

• currently, the paths where Brython searches modules is the list [<brython-path>/Lib, <brython-path>/libs, <script-path>, <brython-path>/site-packages] where <brython-path> is the path to the directory with brython.js and <script-path> is the path to the directory of the currently executing script

• an additional option "cpython_site_packages" can be added to the arguments of brython(). If set to true, the item /cpython_site_packages is added to the list above. If set to "replace", the item <brython-path>/site-packages is removed from the list

• the built-in web server server.py is modified so that when a request starts with "/cpython_site_packages", it searches the file in the CPython site-packages directory

I tested this code with the module urllib3 which is installed by pip on my PC but not in Brython site-packages:

<body onload="brython({cpython_site_packages: 'replace'})">

<script type="text/python">
import urllib3
</script>

Does this solve the issue ? Do you have comments on the implementation ?

For the moment, the built-in server is not included in brython-cli, it is only available if you clone the Github repo. Do you think it would be useful to include it ?

[BrenBarn]

Thanks, Pierre, the cpython_site_packages sounds like basically what I was asking for.

With regard to the implementation, it might be better not to hardcode /Lib/site-packages as you have done. Instead there are ways to have Python tell you where the appropriate directory is. I had been using site.getsitepackages() but this StackOverflow answer suggests that sysconfig.get_path('purelib') might be even better.

I think it would be cool if the server were included in brython-cli. Many other web frameworks include a simple dev server. We've also seen many times on the mailing list that people are confused by the limitations of the file:// protocol. If a server were included this would provide an easy answer to such problems ("run python -m brython --server"). Is there a reason the server hasn't been included as part of the distribution until now?

[PierreQuentel]

In commit 2a2b0cf I have added argument --server PORT to brython-cli. It starts a local web server, based on http.server, mapping requests starting with cpython_site_packages to the path determined by sysconfig.get_path("purelib") as you suggested.

To test it you need to run

python setup.py sdist
python setup.py install

from the setup directory.

If it's ok for you I will add documentation for this feature.

[BrenBarn]

When I try to run setup.py sdist I receive an error that it cannot find the file www/src/brython_no_static.js. What is that file and is it supposed to be there?

[PierreQuentel]

My bad... TL;DR: you have to run scripts/make_dist before setup.py.

Long version : the default brython.js includes the a mapping <stdlib module name>: <address> for the files in the standard distribution, so that when brython_stdlib.js is not loaded in the page, the stdlib files are found with Ajax calls at the address in this mapping.

In the version of brython.js included in the CPython package generated by setup.py (called brython_no_static.js) this mapping is set to {} because the stdlib files can only be imported if brython_stdlib.js is loaded. This version is generated by scripts/make_dist.py.

The commit referenced above prints a message inviting the user to run make_dist if necessary.

[BrenBarn]

Now I get a syntax error:

Traceback (most recent call last):
  File "make_dist.py", line 112, in <module>
    run()
  File "make_dist.py", line 108, in run
    make_VFS.process(os.path.join(pdir, 'www', 'src', 'brython_stdlib.js'))
  File "C:\TheUsers\BrenBarn\Documents\Python\extensions\brython\scripts\make_VFS.py", line 112, in process
    tree = ast.parse(f.read())
  File "C:\FakeProgs\Anaconda3\envs\localbrython\lib\ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 380
    def _create_cb_wrapper(callback, /, *args, **kwds):
                                     ^
SyntaxError: invalid syntax

It looks Brython is trying to compile something with positional-only arguments, which were only added in Python 3.8. Is Python 3.8 required for make_dist? (And is that documented somewhere?)

[PierreQuentel]

Among other things, make_dist.py generates brython_stdlib.js. For each script in the stdlib, the program generates the list of modules it needs to import (this is required in Brython's import mechanism with indexedDB) and for that, it has to parse the script with the ast module; this fails if the CPython version used to run make_dist.py is not at least the one used in Brython stdlib.

In the commit referenced above I have added a test on the version number.

[BrenBarn]

Okay, but where is it documented which version is needed (i.e., what the version of the Brython stdlib is)?

[BrenBarn]

Okay, I got it working, but there is still a problem. I knew there was something else bothering me about the implementation and now I realized what it was. Sorry, I got sidetracked from this issue and forgot what I already said above. (I should have reread my own description! :-)

The thing is that I don't think we actually want Brython to just look in site-packages at all. What we want it to do is actually find the module using the normal Python import process, and then load that file from wherever it may be. This would mean it will also work with .PTH files, custom additions to site-packages, etc. In my original issue description I mentioned this at the end:

That is that the JS end of Brython could have a "dev mode" for its import system. In this mode, in addition to trying the usual requests to find a library, it would also try something like /get_local_library?libname=somelib. In other words, rather than asking for a specific filename, it would just pass the name of the library it's trying to import.

In other words, if my Brython code in the webpage does something like import somepackage.somelib, I would like Brython to not make a request for cpython_site_packages/somepackage/somelib.py, but rather directly pass the "library name" somepackage.somelib (perhaps in a request for something like /cpython_import?library=somepackage.somelib), without trying to "resolve" it to an actual filename. Then the server code would do something like this:

spec = importlib.util.find_spec(module_name)
filename = spec.origin
# now serve up the contents of that filename

Does that make sense? The way it is now, Brython is being very specific about asking for a file, but what I'm suggesting is that it instead (or rather, in addition, as another step in the "search path") ask for a library, because that will give the server the flexibility of the normal Python import mechanism, which can find libraries in many places, not just site-packages.

As a bonus, brython --modules could even do this as well, to actually add the libraries to the VFS, eliminating the need for --add_package (because it would add third-party libs as well as stdlib files).

Thanks for being so responsive on this! I'm willing to do some of the work on the server side if you'd like, I just dare not touch the JS. (That's why I love Brython!)