Investigate python-build-standalone #22

simonw · 2021-08-30T20:32:52Z

https://python-build-standalone.readthedocs.io/en/latest/building.html#macos

Since I would ideally like Datasette.app to support Python plugins, maybe what I really need is a full-blown Python system tucked away inside the Datasette.app macOS package, with the ability to install extra packages into it?

if so, python-build-standalone could be relevant - it's part of the PyOxidizer project.

Maybe I don't need any of the mechanisms from things like PyInstaller or PyOxidizer that combine my code together into a single executable - I just need to be sure that when a user installs the Datasette.app application it has a guaranteed Python environment that it can use to run datasette and its plugins.

The text was updated successfully, but these errors were encountered:

simonw · 2021-08-30T20:33:37Z

I followed the https://python-build-standalone.readthedocs.io/en/latest/building.html#macos instructions and I'm now running the build - it's taken several minutes so far. Looks like it will take a while to finish.

I ran this:

git clone https://github.com/indygreg/python-build-standalone
cd python-build-standalone
./build-macos.py

simonw · 2021-08-30T20:51:13Z

Rather than waiting for it to compile, I decided to try a build from https://github.com/indygreg/python-build-standalone/releases/tag/20210724

Looks like these are the likely options:

cpython-3.9.6-aarch64-apple-darwin-debug-20210724T1424.tar.zst 18.8 MB

cpython-3.9.6-aarch64-apple-darwin-install_only-20210724T1424.tar.gz 38.3 MB

cpython-3.9.6-aarch64-apple-darwin-lto-20210724T1424.tar.zst 28.8 MB

cpython-3.9.6-aarch64-apple-darwin-pgo+lto-20210724T1424.tar.zst 31.3 MB

cpython-3.9.6-aarch64-apple-darwin-pgo-20210724T1424.tar.zst 21.5 MB

I have no idea what the suffixes such as pgo or lto mean.

simonw · 2021-08-30T20:51:45Z

There are now various *-install_only-*.tar.gz release artifacts containing a gzipped tar archive of just the Python installation. These artifacts can be more readily consumed by tools just wanting to run a Python interpreter.

I'll try that one.

simonw · 2021-08-30T20:57:06Z

cd /tmp
mkdir p
cd p
wget https://github.com/indygreg/python-build-standalone/releases/download/20210724/cpython-3.9.6-aarch64-apple-darwin-install_only-20210724T1424.tar.gz
tar -xzvf cpython-3.9.6-aarch64-apple-darwin-install_only-20210724T1424.tar.gz

This gave me a /tmp/p/python folder.

% /tmp/p/python/bin/python3
zsh: bad CPU type in executable: /tmp/p/python/bin/python3

I'll try https://github.com/indygreg/python-build-standalone/releases/download/20210724/cpython-3.9.6-x86_64-apple-darwin-install_only-20210724T1424.tar.gz instead (after rm -rf /tmp/p):

cd /tmp
mkdir p
cd p
wget https://github.com/indygreg/python-build-standalone/releases/download/20210724/cpython-3.9.6-x86_64-apple-darwin-install_only-20210724T1424.tar.gz
tar -xzvf cpython-3.9.6-x86_64-apple-darwin-install_only-20210724T1424.tar.gz

# python/bin/python3
Python 3.9.6 (default, Jul 24 2021, 22:49:46) 
[Clang 12.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

simonw · 2021-08-30T20:57:42Z

It's pretty big unzipped though:

p % du -h python
108M	python

simonw · 2021-08-30T21:01:28Z

I then ran:

% python/bin/pip install datasette==0.59a2
% time python/bin/datasette --version
datasette, version 0.59a2
python/bin/datasette --version  1.08s user 0.13s system 78% cpu 1.546 total

Does this survive being moved elsewhere on the filesystem?

No it does not:

/tmp % mv p/python /tmp/datasette-python
/tmp % cd /tmp/datasette-python 
datasette-python % ls
bin	include	lib	share
datasette-python % bin/datasette --version
zsh: bin/datasette: bad interpreter: /private/tmp/p/python/bin/python3.9: no such file or directory

simonw · 2021-08-30T21:02:50Z

But this DOES work:

datasette-python % bin/python3 -c "from datasette.cli import cli; cli()" -p 8004
INFO:     Started server process [55170]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8004 (Press CTRL+C to quit)

simonw · 2021-08-30T21:05:03Z

This is an interesting option then: it looks like I can create a standalone Python environment using this which could, potentially, be bundled up as part of the Electron Datasette.app folder and used to execute that Datasette server - and since it's a "real" Python environment using pip install to install plugins may work too.

The downside is that it's BIG: 100+MB of disk space.

I can reduce that down a bit though:

% du -h | grep M
1.1M	./include/python3.9
1.1M	./include
1.7M	./lib/python3.9/encodings
1.3M	./lib/python3.9/distutils
4.5M	./lib/python3.9/test/decimaltestdata
 23M	./lib/python3.9/test
1.6M	./lib/python3.9/site-packages/pint
2.2M	./lib/python3.9/site-packages/setuptools-57.4.0-py3.9.egg/setuptools
3.1M	./lib/python3.9/site-packages/setuptools-57.4.0-py3.9.egg
 28K	./lib/python3.9/site-packages/PyYAML-5.4.1.dist-info
1.2M	./lib/python3.9/site-packages/datasette
 24K	./lib/python3.9/site-packages/MarkupSafe-2.0.1.dist-info
2.0M	./lib/python3.9/site-packages/pip-21.1.3-py3.9.egg/pip/_internal
1.5M	./lib/python3.9/site-packages/pip-21.1.3-py3.9.egg/pip/_vendor/chardet
1.0M	./lib/python3.9/site-packages/pip-21.1.3-py3.9.egg/pip/_vendor/distlib
7.0M	./lib/python3.9/site-packages/pip-21.1.3-py3.9.egg/pip/_vendor
9.0M	./lib/python3.9/site-packages/pip-21.1.3-py3.9.egg/pip
9.0M	./lib/python3.9/site-packages/pip-21.1.3-py3.9.egg
 24M	./lib/python3.9/site-packages
2.1M	./lib/python3.9/__pycache__
2.2M	./lib/python3.9/ensurepip/_bundled
2.2M	./lib/python3.9/ensurepip
 35M	./lib/python3.9/config-3.9-darwin
1.8M	./lib/python3.9/idlelib
104M	./lib/python3.9
1.8M	./lib/tk8.6
1.6M	./lib/tcl8.6/encoding
2.4M	./lib/tcl8.6
1.5M	./lib/Tix8.4.3
126M	./lib
127M	.

Looks like I can save 23MB by deleting ./lib/python3.9/test - which hopefully wouldn't break anything else.

I'm loathe to strip out parts of the standard library though since future Datasette plugins might depend on them.

simonw · 2021-08-30T21:08:02Z

Having pip install add files to the folder in /Applications/Datasette.app/... feels wrong to me too. Where does VS Code install its extensions?

Answer: on macOS it stashes them in ~/.vscode/extensions - which on my machine is already 1GB!

simonw · 2021-08-30T21:31:08Z

This works:

mkdir ~/.datasette
/tmp/datasette-python/bin/pip install datasette==0.59a2 datasette-vega -t ~/.datasette --upgrade

And then to run it:

PYTHONPATH=~/.datasette /tmp/datasette-python/bin/python3 -m datasette --version                       
python -m datasette, version 0.59a2
PYTHONPATH=~/.datasette /tmp/datasette-python/bin/python3 -m datasette plugins
[
    {
        "name": "datasette-vega",
        "static": true,
        "templates": false,
        "version": "0.6.2",
        "hooks": [
            "extra_css_urls",
            "extra_js_urls"
        ]
    }
]

simonw · 2021-08-30T21:33:23Z

So... one option would be to bundle standalone Python with the Electron app and then use its pip to install Datasette and associated plugins to a ~/.datasette-app folder.

This would support both plugin installations AND upgrading Datasette itself!

simonw · 2021-08-30T21:38:20Z

Are there any other apps on my machine that are doing that kind of thing already?

find /Applications | grep python | grep -v Xcode

Shows me that both Docker.app and Dropbox.app seem to bundle their own Python: /Applications/Dropbox.app/Contents/Frameworks/libpython3.8.dylib and /Applications/Docker.app/Contents/Resources/bin/docker-compose-v1/resource.cpython-39-darwin.so

Maybe OBS too? /Applications/OBS.app/Contents/MacOS/_obspython.so.

simonw · 2021-08-31T20:30:32Z

I am going to try this:

Bundle a copy of standalone python inside the application folder itself
On the first start up, create a virtual environment in ~/.datasette.app/venv using python -m venv with that bundles Python
Run bin/pip install datasette in that virtual environment

simonw · 2021-08-31T22:09:17Z

I'm going to commit to this mechanism, at least for the moment.

simonw · 2021-09-08T23:42:01Z

Wrote this up as a TIL: https://til.simonwillison.net/electron/python-inside-electron - it's also mentioned in https://simonwillison.net/2021/Sep/8/datasette-desktop/

simonw added packaging Anything involving making stuff installable research labels Aug 30, 2021

This was referenced Aug 31, 2021

Show a loading indicator that isn't a blank screen #23

Closed

Try pyoxidizer as an alternative to PyInstaller #21

Closed

simonw closed this as completed Aug 31, 2021

simonw mentioned this issue Aug 31, 2021

Bundle Python, create virtual environment and install datasette on startup #25

Closed

SimonBiggs mentioned this issue Oct 8, 2021

Building a PyOxidizer binary install of PyMedPhys pymedphys/pymedphys#1556

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate python-build-standalone #22

Investigate python-build-standalone #22

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 31, 2021

simonw commented Aug 31, 2021

simonw commented Sep 8, 2021

Investigate python-build-standalone #22

Investigate python-build-standalone #22

Comments

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 • edited Loading

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 • edited Loading

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 • edited Loading

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 • edited Loading

simonw commented Aug 30, 2021 • edited Loading

simonw commented Aug 30, 2021 • edited Loading

simonw commented Aug 30, 2021

simonw commented Aug 30, 2021 • edited Loading

simonw commented Aug 31, 2021

simonw commented Aug 31, 2021

simonw commented Sep 8, 2021

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021 •

edited

Loading

simonw commented Aug 30, 2021 •

edited

Loading