newbie questions #2
Comments
|
Thanks for your interest in this project. I suggest you go through the README carefully since it directly addresses many topics raised in your questions. First, it's important to understand that this repo is a kind of umbrella project for different ways to use Python in Max/MSP. So in many cases, the answer is qualified by a 'it depends'.
Hope the above was useful. |
|
Hi @omarcostahamido just in case you were not aware I made a pre-release of a couple of python3 externals. Let me know if they work for you. |
|
|
Ok, it looks like there's still a dependency on brew install gettextthen try again. Let me know if you still have problems. |
|
@omarcostahamido, I fixed the dependency issue on the pre-release0 externals (it's is now statically linked at build-time instead of dynamically linked so it should not require the If you have a chance download the new externals from the |
|
Hi @shakfu |
|
btw, having a full python 3.9 system running in that little 9.3 MB external is a pretty impressive feat. I've previously compiled my python as an executable using pyinstaller for quick access (and sharing with non tech-avy users). If it is really possible to compile this guy with custom python packages included I am definitely gonna relearn this and follow your steps :) |
|
Good to hear that you go at least one of the externals working. It's my bad, I should have included a README in the release: the other one, Unlike a regular c external such as
If you look at the image of a minimal package with both externals. The A demonstration of this is give below on the right side of the
Hope all of the above makes sense. My advice is to just use the |
(-: Yes, that was one of the biggest hurdles to making this feasible. In fact packaging was a huge pita which took 3 times as long as development due to a lot of trial and error as there's no official documentation to help you make a minimal python distribution and because Apple's code-signing and notarization rules kept classifying bundled externals which embedded python as illformed after compilation. The Aha moment was discovering that the only place you could keep the python standard lib was in the In any case, if you want to learn how a minimal python distribution is built have a look at the builder python package in the |
|
Hi @shakfu ! Should we be concerned? On the readme you write
There is no pyjs help patch, and I think it would be good to have one since it appears as an object in the package but in reality we are never meant to create a edit: I just suggested a simple help patch on #4 this will be helpful if someone creates a pyjs object and starts scratching their head trying to understand how to use it. I think, for maxers, one of the first instincts is to alt-click to open the help patch 😉 |
|
I would also like to suggest moving all positional arguments into attributes. There are some options that I would like to try but I don't want to declare all the others before getting into that place. And I know that you can use any positional attribute and turn into a sort of attribute just by using its name but you need to know their exact name in order to do that. And if you do that you loose not only the neat autocomplete but also the handy instructions that let you quickly confirm what you are doing without the need to go open its reference page. For example: edit: I just added #5 that address this |
|
Hi Omar, Thanks for your feedback and for revisiting this project. I'll address a number of a your questions in the sequence that I read them: 1. Every time I open py_overview.maxpat there is a some debug information in the console. Should we be concerned? It looks like I left @debug=on in this patch py_overview.maxpat.zip and it further must have cached some paths to related on my system in the patch. You should be able to switch it off by setting @debug=off (although, it seems some logging is not conditioned on the debug flag it seems). In any case, I have fixed this on the main branch (also attached in this post). 2. There is no pyjs help patch, and I think it would be good to have one. An omission on my part. Couldn't agree with you more. The 3. I would also like to suggest moving all positional arguments into attributes. Agreed. Thanks for the pull requests, Omar. Your feedback and contributions are much appreciated! S |
|
Dear Shakeeb, My pleasure. I have my eyes on your repo, and I am really glad you continue to work on it.
But I am just thinking right now if there is a way to programmatically download a python package (we can't assume pip is available right?... or can Question: which one is this patch? If you also don't remember we can recreate it. It should be easily available to users when they open your package. More serious question: do you plan on working on a windows version? I have access to a windows machine with visual studio. I can try dipping my toes here too. Reason for asking this ties back into what I said above about packages. Using the Resources folder is a neat trick but... it is so mac-centered: what happens when you have a windows version for this? It be nice to start thinking about a cross-platform way to go about this. (please correct me if I am missing something). Thx. OCH |
|
I think I need your insight into this. I still can't import python packages. Tried both playing around with
Now, for sake of completion I should note that I myself am running python 3.7.3 on purpose (no fancy homebrew install, just plain old install from python.org) and I wonder if that could influence the version of numpy that I install by default with pip to something that is not compatible with your embedded python 3.9.6 version (?). Thoughts? edit: it seems this problem is specific to NumPy. I just tried another random package and it didn't complain when importing. |
|
Hi Omar, Not sure why, but I'm unable to In the meantime, I'll try to answer your questions as far as I can: How to extend python with your own scripts and 3rd party libraries
For A, I have tested pure python scripts which should work without re-codesigning the externals, but if you add compiled extensions, then I think you have to re-codesign the external. Check out my maxutils project for help with that. For B, this is just a location that's automatically added to the PYTHONPATH, if I recall correctly, so pure python scripts and compiled extensions should be fine for Package Distribution. (code signing may be required for 3rd party distributions) For C, this is just setting that is done at the patch level so it should be straightforward. Note that it is possible to compile a python externals with an embedded pip (see the python Getting numpy to work in The easiest way is to use (1) above and just create an adhoc python external linked to your system python3 setup. If you have numpy installed there, then you should be good to go with the following caveat: the type translation system does not currently automatically cover native numpy dtypes so they would have to be converted to normal lists before use with Max. This is not a hard constraint, just not implemented yet. If you want to go the way of (2) above, then it should be possible to package numpy in the full relocatable external version, but it will take some work. It'll be easiest to just start with the full external build (using the the python Windows version? It should be theoretically possible to make the code more portable for use in windows, but I really gave up on windows development ages ago so it'll have to be someone other than me who make this possible. Where is this patch?
It the S |
|
Hi @shakfu
Strange... I use github desktop it works pretty well.
Understood.
I understand that is the most stable approach, but for the people I work with they'll just loose interest the moment I ask them to open the terminal window or install python. So, yes, I really need the self contained version, which I also think is what makes your project so unique (and I haven't got into your other python implementations yet) Getting numpy to work in I do want to go the route (2). I'm surprised you said:
In my mind the current minimal python external should have no problem accessing it using the PYTHONPATH property to add a directory where the 90mb numpy site-packages folders are located. In fact, that worked just fine to import some other packages just not numpy. Can you try this on your end? Using your current Thank you so much for your thorough response answering all my questions. I really appreciate it! OCH edit: I've noticed that you were able to merge the PRs (thank you), I wonder if you can overcome your issues the same way: creating a fork and sending PRs to here (silly workaround for a silly bug indeed). |
|
Hi @omar With regards to your question about numpy, as you have seen it is not so straightforward to use it for some reason outside of method (1). I will look into t a bit more. There may be a bug in relation the setting of pythonpaths or perhaps the minimal relocatable external is missing something or the other. That's why I had the sense that the regular full build would most likely work. |
|
Fyi Github support fixed the issue with the repo which looked like it was a malfunctioning server from their side. All good now. |
Indeed. But we must push through 😄 I got a hold of a machine where I was able to install python 3.9.6 (from python.org though) and create a virtual env and install numpy, then I grabbed the sitepackages folder and put it on my (3.7.3) venv folder aaaand...
I have been having trouble with pythonpath method. The only way I can make it to work is by setting it on the object itself Thank you for also looking into this. edit: yup right now this is the bottleneck I am facing. I think I tried all the combinations. putting these numpy site-packages folders inside the external resources doesn't help either. |
|
Good to know Github got back to you and fixed problem! Thank you @github 🎉 |
|
Incidentally, I created a 'fat' static python external and installed In the meantime, use method (1) or ... use an alternative packaging variation (it's in the README): In the root of the make -C source/py bin-homebrew-pkgor in make bin-homebrew-pkgNOTE that this will create a In this package, you will find two externals in the virtualenv venv
source venv/bin/activate
pip install numpy
mv venv/lib/python3.9/site-packages/numpy ~/Documents/Max 8/Packages/py/support/python3.9/lib/python3.9/site-packages/numpyThen open
Note that recent changes in Max have allowed for this to work in standalones too. Just create your standalone application from a patcher which includes the |
Ok, trying to use the |
|
I have regular Python (from Python.org). And it is here |
|
So, I know this is terribly messy, but I am sharing here in any case:
I think this all goes to say that I am really trying to get this working, but I am still not entirely sure how to properly use/configure it for my setup (python 3.7.3, installed via python.org installer). If you can make some sense out of this (and I am pretty sure you're the only person who can) it would be great. Thank you! |
|
Hi @omar
Really really appreciate that you are trying to get this working. But first thing you should do is just take a breather and get some perspective on what should be done. On MacOS, in this particular project, I spent the initial 25% of my time happily coding and solving programming problems, and the rest trying to overcome packaging problems (which include codesigning and notarization requirements by Apple). This is the reason why I have developed so many ways to package an external for my particular case (a Homebrew python user). On MacOS Catalina (my current macOS version), there's system python2 ( All of the Please note that I did not test any of my packaging methods on a binary-installed python3 from python.org on a macOS. I have only developed externals that were built from source downloaded from python.org. Indeed, the python package It is a recipe for pain and suffering to try to retrofit one method on the other in the manner in which you were doing earlier. I know this too well (-: In any case, it should not be too difficult to create a method to handle the case of someone who has installed python3 in binary form from python.org: First step is to get the PYTHON_VERSION=$(python3 --version | sed s/Python[[:space:]]//) # 3.9.5
PYTHON_VER=${PYTHON_VERSION%.*} # 3.9
CFLAGS=`python3-config --cflags`
LDFLAGS=`python3-config --ldflags`Please make sure that these are from you python.org installation and not the system pythons installation. Are you able to you obtain these? |
This is a very exciting project, and I am also learning a lot from it, so thank you!
I am also on MacOS Catalina (10.15.7 more precisely)! I have a system python2 but not a system python3 (edit: correction, I do have a python3 in there, but whenever I do
Ok, so I get that what you are trying to say is that I should create a new
Ok, so this is the first thing I really need to clarify. The files I modified on the second bullet point of my last post (
ooof! haha. no need to say more. I know, you know, we all know. But I am still learning this project though! Please have a little patience with me. And thank you for guiding me!
Alright, here we go! PYTHON_VERSION=$(python3 --version | sed s/Python[[:space:]]//)
PYTHON_VER=${PYTHON_VERSION%.*}
CFLAGS=
LDFLAGS=
|
hold on, I'm answering all the things in order. It takes time to do all the compilations here.
Here's what happened: the log 3.7.3.zip Best |
|
After reading through one of your logs, it looks like you are running things out of a virtualenv called MYENV. If so, that may be the problem, since the test environment which was used did not entail using a virtualenv. Please see the attached log of my python3.7 run and where it differs from yours. |
|
After spending more time with your logs, I am more and more convinced that we should put aside Just to highlight the 'breaking' errors in the simpler ( static-ext This error typically indicates missing dependencies during compilation of python. shared-ext Same dependency error as framework-ext In this case, you are unable to build any of the python builtin extensions starting with Here I can see that you have possibly installed a virtualenv So before I spend anymore time trying to solve issues on my front, I think we should establish whether your can build python3.7 from source on your system: Here are basic steps test this:
Please download https://www.python.org/ftp/python/3.7.13/Python-3.7.13.tgz
If you have the dependencies, the you should be able to build the above without errors and then run the built python interpreter without errors: Can you please test this case. Thanks. |
|
Hi @shakfu a edit:
I am running a venv, correct. No I am on Catalina. Never had Mavericks. |
|
Thanks, @omarcostahamido How are you setting up your virtualenv? Have you tried without a virtualenv? |
|
@omarcostahamido May I suggest that you try again without a virtualenv with the 'dev' branch: |
|
Hi @shakfu
that's about it.
Done it on another machine I got access to running macOS 11.6.4 and python 3.7.3 (no virtual env). Cloned your dev branch, ran |
|
Hey @shakfu So, I just realized that you created this folder on my |
|
Not sure I understand why you keep introducing new variables into the test: a virtualenv, or a new computer with a different setup than yours. Indeed the default log which you attached for the latter gives some indication that it doesn't even have Xcode installed. The only way this process is going to work is if one maintains consistent testing conditions. Let's take a pause please. I'm trying to help you solve your specific case provided it remains your specific case. Clearly (from your use of a That's fine. I understand, but let's then forget about the 'dev' branch, and please switch to the master branch of Please cd into |
|
As you perhaps gathered from my last post, I was a little bit frustrated with the last results on your machine and was expressing it in a manner which was, in retrospect, unfair to you. It really was no problem to run out of virtualenv or try another computer to get some additional information. It's on me if the code is not working! In any case, I retested all versions again on my machine, and got positive results. I retested with a virtualenv and got the same. To remove the possibility that Homebrew on my machine was giving me a hidden advantage with respect to dependencies, I tried another machine (an older MacBook air, with Catalina) and removed all traces of Homebrew on it and got positive results. In all cases, I ran the From one perspective, it is also positive outcome that you were able to compile python to the end without error. If this is reproducible, then the errors may be coming from some of my post-processing code (to shrink python basically), so another testing scenario could be that I create a Also taking a few steps back from trees, it should be unnecessary for you, and perhaps others having the same issues as you, to compile the externals and install xcode if we have automated builds which are producing and releasing build variants in a standardized way (via Github Actions for example). Then you just install the package and you are off. This is another area where a lot can be done. So let's keep going. |
|
The 'vanilla' variations, mentioned in my previous post have been pushed to 'dev' branch. These build variations have no cleaning or post-processing except for 'tests' removal and removal of *.pyc files. This alone shrinks each external from 150 MB to 37 MB I'd appreciate if you can try both versions:
make vanilla-extand
make vanilla-pkgI hope these work for you and thanks again for your help! |
|
Hi @shakfu I'm sorry the 11.6.4 tests were not helpful at all. I had limited time with that machine and totally forgot to check xcode 😞
I'd like to test that yes. I'm baffled that you copied my setup and got different results! I really have no idea what could be going on here... should we check the xcode version? I'm running xcode 11.6 here.
Wait, what!? You can setup github actions to build max externals?? I am very ignorant about github actions but I would very much like to learn a little bit about it. I would think that you need a mac (or windows) system to compile objects for those systems... github actions run on a linux system, no? (I mean, I remember seeing somewhere that you can actually point a local machine to be the designated worker for your action, but at that point it becomes just another layer of local setup in this case..)
Ok, here we go: So, Then hum... edit: Just tried running |
|
Thanks for testing. No worries, even if it's not working right now, the information that you just provided helps towards localizing the problem.
I'm also very curious why it hasn't been working on your system. I don't think it's an issue if you are running Xcode 11.6. I'm running Xcode 12.4 myself (on Catalina 10.15.7). I'm not sure that's the problem.
I'm just starting to learn about Github actions. There's even an approved cycling74 external template which has a pretty decent Github action configuration (for macos/windows) see here I'm going to have a go at getting this working. I thought the codesigning part would be a blocker but apparently there are GitHub actions which do this. So theoretically, I should be able to generate all of the variations (for mac and windows) on demand.
Ok, my bad, I should have asked you to pull the latest 'dev' branch commit to get a fix (avoid downloading dependencies for vanilla builds) and also do this to get logging: make test-vanilla-extI'll attach a log of a successful build of the above, logs shoud be in also try the following: make test-vanilla-pkgThanks! |
|
Hi @shakfu This is the playlist regarding github actions that I was checking out the other day.
🙌
Ok, here it goes: Thank you for looking into this |
|
It's quite curious why your system is failing to compile the external. Let's just look at one case: I have attached three files, your failed log of compiling 4205c4211
< Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
---
> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
4259c4265
< checking for -llvm-profdata... no
---
> checking for llvm-profdata... no
4309,4311c4315,4317
< checking libintl.h usability... no
< checking libintl.h presence... no
< checking for libintl.h... no
---
> checking libintl.h usability... yes
> checking libintl.h presence... yes
> checking for libintl.h... yes
4535c4541
< checking for textdomain in -lintl... no
---
> checking for textdomain in -lintl... yes
4544c4550,4551
< checking for pkg-config... no
---
> checking for pkg-config... /usr/local/bin/pkg-config
> checking pkg-config is at least version 0.9.0... yes
4547a4555I think this is most likely where the failure occurs, your system does not have pkg-config, which is defined as:
> checking for --with-decimal-contextvar... yes
4552c4560
< checking if PTHREAD_SCOPE_SYSTEM is supported... yes
---
> checking if PTHREAD_SCOPE_SYSTEM is supported... no
4572c4580
< checking for bind_textdomain_codeset... no
---
> checking for bind_textdomain_codeset... yes
4839,4840c4847,4848
< checking whether wchar_t is signed... yes
< checking whether wchar_t is usable... no
---
> checking whether wchar_t is signed... no
> checking whether wchar_t is usable... yes
4845c4853
< checking whether right shift extends the sign bit... yes
---
> checking whether right shift extends the sign bit... no
4896c4904
< checking for pkg-config... no
---
> checking for pkg-config... /usr/local/bin/pkg-config
4924c4932
< 47672 INFO : VanillaPythonForExtBuilder make altinstall
---
> 56596 INFO : VanillaPythonForExtBuilder make altinstall
5043c5051
< ./Modules/getpath.c:676:20: warning: 'NSModuleForSymbol' is deprecated: first deprecated in macOS 10.5 - dladdr() [-Wdeprecated-declarations]
---
> ./Modules/getpath.c:672:20: warning: 'NSModuleForSymbol' is deprecated: first deprecated in macOS 10.5 - dladdr() [-Wdeprecated-declarations]
5046c5054
< /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach-o/dyld.h:199:21: note: 'NSModuleForSymbol' has been explicitly marked deprecated here
---
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/mach-o/dyld.h:199:21: note: 'NSModuleForSymbol' has been explicitly marked deprecated here
5049c5057
< ./Modules/getpath.c:676:38: warning: 'NSLookupAndBindSymbol' is deprecated: first deprecated in macOS 10.4 - dlsym() [-Wdeprecated-declarations]
---
> ./Modules/getpath.c:672:38: warning: 'NSLookupAndBindSymbol' is deprecated: first deprecated in macOS 10.4 - dlsym() [-Wdeprecated-declarations]
5052c5060
< /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach-o/dyld.h:189:17: note: 'NSLookupAndBindSymbol' has been explicitly marked deprecated here
---
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/mach-o/dyld.h:189:17: note: 'NSLookupAndBindSymbol' has been explicitly marked deprecated here
5055c5063
< ./Modules/getpath.c:678:27: warning: 'NSLibraryNameForModule' is deprecated: first deprecated in macOS 10.5 [-Wdeprecated-declarations]
---
> ./Modules/getpath.c:674:27: warning: 'NSLibraryNameForModule' is deprecated: first deprecated in macOS 10.5 [-Wdeprecated-declarations]
5058c5066
< /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/mach-o/dyld.h:169:21: note: 'NSLibraryNameForModule' has been explicitly marked deprecated here
---
> /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/mach-o/dyld.h:169:21: note: 'NSLibraryNameForModule' has been explicitly marked deprecated here
5110c5118,5119
< -framework CoreFoundation -ldl -framework CoreFoundation;
---
> -framework CoreFoundation -lintl -ldl -framework CoreFoundation;
> ld: warning: dylib (/usr/local/lib/libintl.dylib) was built for newer macOS version (10.15) than being linked (10.13)
5121c5130
< sed 's/%VERSION%/'"`DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.3 ./python.exe -c 'import platform; print(platform.python_version())'`"'/g' < Mac/Resources/framework/Info.plist > ~/.build_pyjs/lib/Python.framework/Versions/3.7/Resources/Info.plist
---
> sed 's/%VERSION%/'"`DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.9 ./python.exe -c 'import platform; print(platform.python_version())'`"'/g' < Mac/Resources/framework/Info.plist > ~/.build_pyjs/lib/Python.framework/Versions/3.7/Resources/Info.plist
5129c5138,5139
< gcc -Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.7/Python -o python.exe Programs/python.o -ldl -framework CoreFoundation
---
> gcc -Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.7/Python -o python.exe Programs/python.o -lintl -ldl -framework CoreFoundation
> ld: warning: dylib (/usr/local/lib/libintl.dylib) was built for newer macOS version (10.15) than being linked (10.13)
5134c5144
< ./Tools/pythonw.c:214:9: warning: comparison of function 'posix_spawn' not equal to a null pointer is always true [-Wtautological-pointer-compare]
---
> ./Tools/pythonw.c:223:9: warning: comparison of function 'posix_spawn' not equal to a null pointer is always true [-Wtautological-pointer-compare]
5137c5147
< ./Tools/pythonw.c:214:9: note: prefix with the address-of operator to silence this warning
---
> ./Tools/pythonw.c:223:9: note: prefix with the address-of operator to silence this warning
5174c5184
< DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.3 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
---
> DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.9 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
5181c5191
< DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.3 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python.exe -E ./setup.py build
---
> DYLD_FRAMEWORK_PATH=~/.build_pyjs/src/Python-3.7.9 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python.exe -E ./setup.py buildBelow you have the first sign of actual major failure: failing to build 5185,5187c5195,5202
< gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -IObjects -IInclude -IPython -I/usr/local/include -I~/Documents/Max 8/Packages/py-js/source/py/MYENV/include -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _struct.c -o build/temp.macosx-10.9-x86_64-3.7/_struct.o
< clang: error: no such file or directory: '_struct.c'
< clang: error: no input files
---
> creating build/temp.macosx-10.13-x86_64-3.7/Users
> creating build/temp.macosx-10.13-x86_64-3.7~
> creating build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs
> creating build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src
> creating build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src/Python-3.7.9
> creating build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src/Python-3.7.9/Modules
> gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include -I. -I/usr/local/include -I~/.build_pyjs/src/Python-3.7.9/Include -I~/.build_pyjs/src/Python-3.7.9 -c ~/.build_pyjs/src/Python-3.7.9/Modules/_struct.c -o build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src/Python-3.7.9/Modules/_struct.o
> gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.13-x86_64-3.7~/.build_pyjs/src/Python-3.7.9/Modules/_struct.o -L/usr/local/lib -o build/lib.macosx-10.13-x86_64-3.7/_struct.cpython-37m-darwin.so
5189,5192c5204,5206
< creating build/temp.macosx-10.9-x86_64-3.7/_ctypes
< gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I. -IObjects -IInclude -IPython -I/usr/local/include -I~/Documents/Max 8/Packages/py-js/source/py/MYENV/include -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c _ctypes/_ctypes_test.c -o build/temp.macosx-10.9-x86_64-3.7/_ctypes/_ctypes_test.o
< clang: error: no such file or directory: '_ctypes/_ctypes_test.c'
< clang: error: no input files
---After the failure to build So my brief analysis leads me to speculate that the lack of See the pdf of the man page of My advice is have a look where the compilation is taking place in |
|
I have just run my first 'matrix'-type github workflow (i.e. macos 10.15 Catalina across pythons 3.7 to 3.10) and happily, I had a successful run on the first attempt which matches the testing I've been doing on my own machine. The success of this run increases the likelihood that you are missing something or the other which is preventing your machine from building successfully. One thing occurred to me recently: have you tried to update your command-line tools? (see this stackoverflow article) which may help. Incidentally, the successful run was running the 'master' branch, not the 'dev' branch which I have kept stable to test your particular setup preferences. The GitHub workflow used was just a test run which didn't produce anything that one can use. In fact, it's really basic, but I am quite impressed how powerful this system can be: name: Build2
on:
workflow_dispatch:
jobs:
build:
name: macos-10.15 python-${{ matrix.python-version }}
runs-on: macos-10.15
strategy:
matrix:
python-version: ['3.7', '3.8', '3.9', '3.10']
steps:
- uses: actions/checkout@v3
with:
submodules: 'recursive'
fetch-depth: '0'
- name: setup python
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Display Python version
run: python3 --version
- name: build_default
run: make
- name: build_shared_pkg
run: make shared-pkg
- name: build_shared_ext
run: make shared-ext
- name: build_static_ext
run: make static-ext
- name: build_framework_pkg
run: make framework-pkg
- name: build_framework_ext
run: make framework-ext
- name: build_relocatable_pkg
run: make relocatable-pkgThe next iteration will attempt to try to extract the build products, codesign them, notarize them, and package them for release / download. Before I do that, I want to get Apple Silicon compatibility working. I took a step towards that on the master branch by updating the sdk to version 8.2 and exclusively using So this direction looks promising. |
|
Hi @shakfu I'm really enjoying this thread.
I remember updating the command line tools at some point yes. Also, wow! I'm also really impressed that workflow actions are working! This is really exciting. edit: note to self this is a nice starting place for git actions. edit2: BTW, I was looking at the workflow and wondering if it wouldn't be better to split each build flavour into separate jobs so that they can run in parallel... |
|
Hey @shakfu So, after installing command line tools (again?) I'm now trying again I reckon |
|
Hi @omarcostahamido,
One thing for sure is that it isn't on your system, but it's also the case that it isn't installed by default by Xcode tools, so it may not even be required. I was hoping that an update to Another option would be to install the homebrew base system (without installing homebrew python) which might help. In fact all of the macos build environments used by GitHub have homebrew installed (see macos-10.15 and macos-11
I'm still getting comfortable with the GitHub workflows and actions. I figured out how to cache and how to make available the externals at the end of a run, but I doubt these are usable until they are codesigned and notarized. This is possible but I haven't gotten round to it yet. My current plan after trying a bunch of different configurations is to only use matrix-based multi-os and multi-python-version 'matrix'-based workflows for testing (without trying to collect the produced externals and linked python distributions). It's too complex to do this, and it isn't necessary to have one solution which fits all cases. I am now testing a model which envisions one workflow per variant (using a simple matrix build to generate a package for each python versions). This can be extended in the future with an os dimension (for windows and arm64 architectures) to generate a package for os / python-version or even forgot about python-versions and just do iterate over os cases. Incidentally, Github workflows don't actually support compiling on arm64 for macos yet.. so I can't use this system to test for apple silicon compatibility. I think this will come though. Mostly, likely both GitHub (microsoft) and Apple have an interest in seeing this being made possible. |
|
Hi @shakfu
You always finding a way to rope me back into homebrew... haha
I'd say your current goal is already pretty ambitious, but I like it!
So... you can focus on Windows first! Great! *wink |
|
Just fyi, I've implemented on-demand build (select any version of python3.7 to 3.10), with signing and notarization using GitHub workflows and actions. The workflow in question is called There's a caveat though, for Incidentally, I've been thinking that this extremely long exchange around one issue which you started. This has been great in many ways, but I think because of its length, it is perhaps not helpful for newbies who may get put off by the length and variety of topics discussed. It's probably better to close this and open up a bunch of topical issues which are shorter and more to the point. We can maybe have another issue called general development ideas or something for more general discussions and other one for for updates... I have a couple more things I'd like to finish before trying to solve the compatibility problem (apple silicon or windows), namely producing a 'tiny' python3 external: I have it down to 8.8MB now, I want to keep going and see how far I can go before it becomes useless 😄 |
|
Hi @shakfu Congrats on pulling this off. And just like that I learned about actions artifacts, a neat way to store the output of running an action without having to make a release.
Are the I've also been meaning to ask: in this table you mention that pkg variants have pip, but how do you actually use pip there? Is it possible to request a release asset?
I agree. It is surely one of the longest github issue threads I've been on. And I'll remember it very dearly haha. So, if you must, please close this issue. It's been a great pleasure to have this open 1-on-1 conversation with you, Shakeeb.
Ha! One last thing, this reminded me of what you said previously:
Since you moved sdk 8.2, shouldn't it be possible to compile a m1 compatible external on an intel-based mac? I'm pretty sure I've heard/read somewhere that is a thing. |
|
Let's conclude this thread with some good news.
Good news! This is now resolved, and one can generate on-demand both forms (self-contained externals and externals which depend on a Python.framework in the There was a hiccup along the way which prevented me from reporting this good news earlier, it seems that the
cd support
Python.framework/Versions/3.9/bin/pip3 install <something>or since, you asked, I have uploaded the above and put it into 0.1 release section as a pre-release: py-js-relocatable-pkg-darwin-x86-3.9-5e7754b.zip
I don't think i can do that on my intel mac running Xcode 12 on Catalina. No matter, I do have an M1 MacBook air which should help going forward. |
|
Hi @shakfu Great news indeed! I very eagerly downloaded the py-js-relocatable-pkg-darwin asset and copied over the externals and support folder from the dmg file. When trying to install anything, though, I am faced with: $ Versions/3.9/bin/pip3 install termcolor
Collecting termcolor
Using cached termcolor-1.1.0.tar.gz (3.9 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.edit: and for the record $ Versions/3.9/bin/pip3 list
Package Version
---------- -------
pip 22.0.4
setuptools 58.1.0
wheel 0.37.1 |
That's what you get for playing with a pre-release 😄 Ok, before I uploaded it, I pip installed It seems I may need to reduce the aggression on the shrinking I apply to relocatable: I was able to upgrade
So that's the mystery of termcolor, or rather of setup tools needing ctypes which was removed.....By the way termcolor has no place in a python3 embedded in Max!! I think the moral of the story is to release some kind of vanilla-relocatable-pkg which has minimal shrinking steps and maximum compatibility. |
|
Hi @shakfu
yes, I know 😄 I just tried to install a couple different small python packages and they all failed with that same message. |
|
I have a fix for pip-bug in Again, it's in the 0.1 release section under py-js-relocatable-pkg-darwin-x86-3.9-688a4ba.zip Hoping this one actually works for you. |
|
Hi @shakfu
haha 😆 Ok, let's see the new version: $ Versions/3.9/bin/pip3 install pitchtools
Collecting pitchtools
Using cached pitchtools-1.5.tar.gz (13 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: pitchtools
Building wheel for pitchtools (setup.py) ... done
Created wheel for pitchtools: filename=pitchtools-1.5-py3-none-any.whl size=13087 sha256=64f64ce7d0a4b823f23106f9e817eb27a42b808933a34f7d002b43b197741140
Stored in directory: /Users/omar/Library/Caches/pip/wheels/41/0d/56/c63ebf95e73004298965bf0eda2add8441b22b1ad55c8cd8ff
Successfully built pitchtools
Installing collected packages: pitchtools
Successfully installed pitchtools-1.5It worked! 🎉 I'm a bit sad to say it, but this probably means that this thread is coming to a close, as you've completely addressed (in more ways than anyone could have anticipated) all the issues from the original post. This ability to work with the external packages was indeed the last point on that post! So, I'm gonna miss our interactions here (*) but you should feel free to close this issue now. Thank you so much. It's been a pleasure. OCH (*) unless you keep tagging me somewhere else to share some good news about this project 😄 |
|
That's great, Omar. I'm pleased it finally worked for you. Thank you very much for your feedback throughout. Tt definitely spurred me forward, and kept me on my toes 😄 |
Hi @shakfu
What a great repo!
I have some newbie questions if you don't mind me asking:
Thank you for your attention and sorry for the long list of questions.
Best,
OCH
The text was updated successfully, but these errors were encountered: