Embed Max python3 external

[shakfu]

Cool project, I think you are the first Max project which may actually have a use for one of my python3 externals for Max

It looks like you are just using a python client to contact, in that case you can easily embed the client n your Python3 external and then package your Max patch as a Max Package or a Standalone. In any case, check out the project. It even has code signed and notarized releases. (-:

One big caveat though: it's macOS only.

[hugofloresgarcia]

oh sweet! i was looking for something like this to make setup easier! I'll take a look and see if I can build it into this!

[shakfu]

Unless you have a strong preference to compile everything from source, I think the quickest way for you to get started is just to use one of recent build variants from the releases section.

There are a bunch which serve different deployment purposes (the README helps here), but I will just cover the simplest, the self-contained external variants (such 'shared-ext', 'static-ext' or any of the 'tiny-variants:

• If you are running just pure python code then it's very simple and just open the bundle and drop the python module or package into the site-packages folder inside the external itself: py.mxo/Contents/Resources/lib/python3.11/site-packages. Then you are good to go..

• If you are running pure python + compiled extensions, then it gets a bit more complicated due to Apple's code signing requirements and you will have to re-sign (and possibly re-notarize) the external bundle yourself if you want to distribute widely. There's code to do that automatically in the project if you have an apple developer account) or just ask your users to sudo xattr -r -d com.apple.quarantine <external.mxo> from the command-line.

[hugofloresgarcia]

I'm trying to work with static-ext, and want to embed the vamp package in this repo, which depends on gradio-client and argbind. It doesn't look like these packages use compiled extensions, so I think I should be able to drop the package into site-packages. I'm assuming I'd have to copy the packages for vamp, gradio-client, argbind, and all their dependencies right? should I just copy over the site-packages from the local python environment from which I'm working with?

[hugofloresgarcia]

Is py-js-static-ext-darwin-arm64-3.11.2.dmg.zip signed? seem to be running into a quarantine issue with it

[hugofloresgarcia]

seem to be running into the same issue with py-js-shared-ext-darwin-x86-3.11-170c83f.zip. Got the releases from here: https://github.com/shakfu/py-js/releases

[shakfu]

I'm not sure why, if you have not modified it any way, it should work. I'll also check on an M1 mac, but on intel MacBook pro I just checked the static-ext version:

$ codesign --check-notarization --verify -v externals/py.mxo/Contents/MacOS/py
externals/py.mxo/Contents/MacOS/py: valid on disk
externals/py.mxo/Contents/MacOS/py: satisfies its Designated Requirement

[shakfu]

Here's the same from an M1 MacBook:

% uname -a
Darwin minx 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64
% codesign -v --check-notarization --verify py.mxo/Contents/MacOS/py
py.mxo/Contents/MacOS/py: valid on disk
py.mxo/Contents/MacOS/py: satisfies its Designated Requirement

[hugofloresgarcia]

ah! my bad, seems like adding dropping in the packages in site-packages had messed up the code signature. If I delete the packages I added, I can load py w/o problem.

[hugofloresgarcia]

gonna try building from source!

[shakfu]

ah! my bad, seems like adding dropping in the packages in site-packages had messed up the code signature. If I delete the packages I added, I can load py w/o problem.

Well, at least you have a working baseline. Incidentally, I checked the dependencies of vamp on a virtualenv and I think I found the culprit:

$ virtualenv venv && source venv/bin/activate
created virtual environment CPython3.11.4.final.0-64 in 500ms
(venv) $ pip install vamp
Collecting vamp
  Using cached vamp-1.1.0-cp311-cp311-macosx_12_0_x86_64.whl
Installing collected packages: vamp
Successfully installed vamp-1.1.0

(venv) $ cd venv/lib/python3.11/site-packages/
(venv) $ ls
__pycache__/                     setuptools/
_distutils_hack/                 setuptools-67.8.0.dist-info/
_virtualenv.pth                  setuptools-67.8.0.virtualenv
_virtualenv.py                   vamp/
distutils-precedence.pth         vamp-1.1.0.dist-info/
pip/                             vampyhost.cpython-311-darwin.so*
pip-23.1.2.dist-info/            wheel/
pip-23.1.2.virtualenv            wheel-0.40.0.dist-info/
pkg_resources/                   wheel-0.40.0.virtualenv
(venv)$ fd --glob  '*.so'
vampyhost.cpython-311-darwin.so

As you can see vamp has a compiled extension vampyhost.cpython-311-darwin.so which will 'break' the pre-existing code signing of the external bundle to give the error you saw. This is expected, and can be fixed by rec-codesigning the bundle (and of course, the binary: vampyhost.cpython-311-darwin.so)

Hope this helps.

[hugofloresgarcia]

Thanks! I've got one last question, is there a way to build py with those packages built in? or is there a more appropriate way of being able to call the script from the py external?

[shakfu]

gonna try building from source!

Feel free to compile as you like (the README is helpful), but you'll end up with the same codesign issue as above.

By the way, can you try the following as an experiment:

1. virtualenv venv
2. `source vent/bin/activate
3. `pip install vamp gradio-cliena argbind
4. copy the relevant modules from the virtualenv site-packages to the (downloaded) signed external's site-package
5. git clone https://github.com/shakfu/py-js
6. put the external which has the copied modules in its site-packages in the externals folder of the newly cloned py-js project.
7. make sign from the root of the cloned project
   and test it.

[shakfu]

I just found a newly introduced bug that will break compilation. I've fixed it now.

[shakfu]

Thanks! I've got one last question, is there a way to build py with those packages built in? or is there a more appropriate way of being able to call the script from the py external?

You can build them in manually as above, or you can set the python path in the external itself (see README).

[hugofloresgarcia]

I tried the former (copying site-packages from a venv and codesigning it from the repo), but it looks like there were packages missing that were not in my site-packages from the virtualenv environment

[hugofloresgarcia]

after reading the readme a bit, maybe what I need is to build a framework-pkg so I can pip install from the external itself?

[shakfu]

Ok, I've done it.
It's large but it includes vamp, gradio-client, argbind and also numpy. It's signed and notarized too. You can download it from my releases section (vamp edition): py-js-shared-ext-darwin-arm64-3.11.4-vamp.dmg.zip

[shakfu]

after reading the readme a bit, maybe what I need is to build a framework-pkg so I can pip install from the external itself?

Yes, that's exactly right, but in my efforts to shrink the externals, I dropped unicodedata and you need it to pip install (if you want pip install), but manual installation works.. until I fix the pip issue.

Anyway, try the external I linked to above and let me know if it works for you somehow.

[shakfu]

@hugofloresgarcia in case you missed it somehow: py-js-shared-ext-darwin-arm64-3.11.4-vamp.dmg.zip imports vamp without errors.

I maybe able to reduce its size a little but numpy is a sizeable dependency.

Have fun.

[hugofloresgarcia]

Ah, thanks so much for helping me figure this out, and getting that binary w/ the libraries in it! Trying it now.

[hugofloresgarcia]

Ah! looks like the vamp package that's in there is actually this, hence the need for numpy I think? The vamp package i referred to is actually the package embedded in this repo. I've renamed the package to unloop avoid that name conflict!

[hugofloresgarcia]

We're you able to create that executable by installing the packages in a separate environment, copying them over to the external, then re-signing the executable? I'd like to be able to figure out how to do it on my end, so I can change the python packages if needed for unloop.

[hugofloresgarcia]

Can't seem to run the unloop entry script with execfile either :(

[hugofloresgarcia]

I tried copying the renamed unloop package to the shared-ext you sent and re-signed it with no problem. When trying to execute though, still running into a ModuleNotFoundError

[shakfu]

After getting some sleep, I had another go and here's the result:

1. py-js-shared-ext-darwin-arm64-3.11.4.dmg.zip - It's a Max package, so you can just grab the external from its externals folder and use it in your own package. It has unloop and its dependencies already include, and I've tested that from unloop.unloop import vamp imports without errors. You can test the same using the py_overview.maxpat file in the patchers folder.

2. DEPS.zip: these are the files that I copied into site-packages folder in the external py,mxoWITHOUT the extensions. I found out that all of the four compiled extensions were optional and just there for performance. So I have tested this set in a framework-pkg build variant whose usefulness you already picked up on: since the the python interpreter is in the support folder and not in the external itself and yet the external is dynamically linked to the same python interpreter, it means that you side-step the code signing issues.

So in summary (1) is a self-contained code signed and notarized external with compiled extensions and (2) is a folder of the same dependencies but with extensions removed and it works well in a framework-pkg build variant, because, in this case, the external that is code signed and hashed, is never changed.

Also there's a third way here: if you want to make changes going forward and must use compiled extensions just create something like the DEPS folder using a virtualenv (until there's a more user-friendly mechanism, that is) and copy the contents into the site-packages of the same external, and then most importantly, put the modified external into the externals folder of the py-js repo that you cloned and then run make sign.

This will codesign the external with an ad-hoc signature and then you can package the result and ship it to users even if you don't have an apple developer subscription with the caveat that there will be a "remove quarantine" dialog on Max the moment it's used. But this is a one time thing, and users are kind of used to it by now.

Of course, you can get rid of that quarantine by getting an apple dev subscription (around 100. USD / year), but in this case, and in the case of open-source software in general, it's clearly not worth the cost.

Have fun.

[shakfu]

@hugofloresgarcia did you have any luck with the last external I attached above?

[hugofloresgarcia]

hey @shakfu! haven't gotten a chance to try it yet, dealing with some other research stuff today. will probably get a chance to try it in a couple hours and i'll let you know!

[shakfu]

Hey @hugofloresgarcia

Not sure you how got along...

But in case you are still interested to embed your python code into a Max package or a standalone. There's a very simple solution now: just make relocatable-pkg in the root of the py-js project.

This will install a relocatable Python.framework in the support directory (it will have pip, so you can install whatever you like into it), it will also build and link a python3 external against this framework. This structure can be used in a Max package or a standalone.

[hugofloresgarcia]

Thanks so much @shakfu!!! I have a deadline coming up on the 31st for another project that's taking a bunch of my time, but planning to get back on this to make this easier to distribute! Thanks a lot for your help!