Trouble reproducing the build

[oflebbe]

Hi I am trying to follow the instructions to build , but somehow it does not work quite well for me :)

I see a lot of messages like

error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: build/lib.macosx-11.1-x86_64-3.9/Frameworks//python3_ios-_cffi_backend.framework/python3_ios-_cffi_backend (No such file or directory)
Creating:  lib.darwin-arm64-3.9/Frameworks/_cffi_backend.framework

While other frameworks seem to be built, the _cffi_backend framework dir is not even created before .

Other problem is that the xcode proj seems to reference binaries from other parts of the filesystem (Folder referenced for ios_system, network_ios, lua_ios) . It seems to come from "../../../../../../../../src/Xcode_iPad/a-Shell" in a-Shell.xcodeproj/project.pbxproj . Am I supposed to built this prior to this project at that specific location ?

[holzschu]

Hi, the frameworks referenced should now be "Relative to build products" with commit e340465, which should solve the second problem.

For the first problem, compiling cffi is always delicate. If you open make_install_osx.log and make_ios.log and look for the line building '_cffi_backend' extension, right after that you should see the error message from the compiler. With that, maybe we can understand the issue and fix it.

[oflebbe]

Ah well ... here it is 👍
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SS L module is not available.")': /simple/pip ...

Seems to be the consequence of

Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().

Should I have installed openssl sources somewhere ?

[oflebbe]

Thanks for the update of the XCode project, btw!

[holzschu]

Ah yes, I remember this issue for other users. Yes, you need OpenSSL installed on your system so that Python (for OSX) uses it when building itself. There is an extra variable EXTRA_CONFIGURE_FLAGS_OSX which is passed to configure and that you can use to specify the location of your SSL library and headers. You don't need the OpenSSL sources, but you need an install (library and headers), and it needs to be in the path that clang is going to use.

[oflebbe]

Understood I am trying my best with
export EXTRA_CONFIGURE_FLAGS_OSX=-with-openssl=/usr/local/Cellar/openssl@1.1/1.1.1i/
now

[oflebbe]

OK next problem:
Got

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.
platform/Developer/SDKs/MacOSX11.1.sdk -I/Users/olaf/Devel/oss/a-shell/cpython/ -I/usr/local/include/ -DCRYPTOGRAPHY_OSRANDOM_ENGINE=CRYPTOGRAPHY_OS
RANDOM_ENGINE_DEV_URANDOM -I/Users/olaf/Devel/oss/a-shell/cpython/Library/include/python3.9 -c build/temp.macosx-11.1-x86_64-3.9/_openssl.c -o build
/temp.macosx-11.1-x86_64-3.9/build/temp.macosx-11.1-x86_64-3.9/_openssl.o -Wconversion -Wno-error=sign-conversion
build/temp.macosx-11.1-x86_64-3.9/_openssl.c:575:10: fatal error: 'openssl/opensslv.h' file not found
#include <openssl/opensslv.h>
         ^~~~~~~~~~~~~~~~~~~~
1 error generated.

Seems you have openssl headers in /usr/local/include , right ?
in buildAllArchitectures.sh
env CC=clang CXX=clang++ CFLAGS="-I$PREFIX/ -I/usr/local/include/ -DCRYPTOGRAPHY_OSRANDOM_ENGINE=CRYPTOGRAPHY_OSRANDOM_ENGINE_DEV_URANDOM" LDFLAGS="-L$PREFIX/ -L/usr/local/lib" python3 setup.py build >> $PREFIX/make_install_osx.log 2>&1

Maybe we should ask everyone to do this as well ?

[holzschu]

I think in the future I will provide OpenSSL as a precompiled framework for OSX (I already provide it for iOS and the simulator, so I only have to add one architecture). In the meantime, your best move is probably to replace this "/usr/local/include" with the directory where your openssl headers are, and same with "/usr/local/lib".

Or you could change the way EXTRA_CONFIGURE_FLAGS_OSX works and an argument for OPENSSL_DIRECTORY, and then forward that directory to the cryptography package (-I $OPENSSL_DIRECTORY/include).

[oflebbe]

Next Problem, sorry :)

Download process is downloading lxml-4.6.2 but trying to unpack 4.6.1

~/Devel/oss/a-shell/cpython/packages ~/Devel/oss/a-shell/cpython
Collecting lxml
  Using cached lxml-4.6.2.tar.gz (3.2 MB)
Saved ./lxml-4.6.2.tar.gz
Successfully downloaded lxml
tar: Error opening archive: Failed to open 'lxml-4.6.1.tar.gz'
./buildAllArchitectures.sh: line 393: pushd: lxml*: No such file or directory
cp: ../setupinfo_lxml.py: No such file or directory
python3: can't open file '/Users/olaf/Devel/oss/a-shell/cpython/packages/setup.py': [Errno 2] No such file or directory

Changing it for now to
tar xvzf lxml-4.6.*.tar.gz

And forgot to link all the ssl libs to /usr/local/lib as well . OMG

[holzschu]

That one is weird, since the relevant lines in the script are:

rm -rf lxml*  >> $PREFIX/make_install_osx.log 2>&1
python3.9 -m pip download --no-binary :all: lxml >> $PREFIX/make_install_osx.log 2>&1
tar xvzf lxml*.tar.gz  >> $PREFIX/make_install_osx.log 2>&1
rm -rf lxml*.tar.gz >> $PREFIX/make_install_osx.log 2>&1
pushd lxml*  >> $PREFIX/make_install_osx.log 2>&1

If you run the script multiple times (as we all do when debugging) there can be several versions of a package, which confuses the script, but that's precisely why there is rm -rf lxml* at the beginning. It seems the rm was not efficient.

[oflebbe]

In my checkout it reads

buildAllArchitectures.sh

...
tar xvzf lxml-4.6.1.tar.gz  >> $PREFIX/make_install_osx.log 2>&1
rm -rf lxml*.tar.gz >> $PREFIX/make_install_osx.log 2>&1
...

[holzschu]

Ah, maybe I did not update recently. Does a git pull in the cpython directory improve things?

[oflebbe]

Next issue

build/temp.macosx-11.1-x86_64-3.9/scratch/vers.c:4:10: fatal error: 'zmq.h' file not found
    #include "zmq.h"
             ^~~~~~~

ZeroMQ ? Really? Should I have this installed as well?

[oflebbe]

OMG was not aware cpython is a submodule ... I was working in a old version.

[holzschu]

ZeroMQ is used by Jupyter. I'm adding it to the list of frameworks where I should make a 3-platform architecture rather than 2, for better portability.

[oflebbe]

No need to provide it, only was expecting to know beforehand what I need to provide (Did I miss to find it somewhere mentioned?) I am actually asking myself if https://github.com/Carthage/Carthage wouldn't be more convenient to support the multiple frameworks on multiple architectures.

[oflebbe]

Now freetype from Homebrew is picked up by the simulator build rather the provided Framework ... Seems the autoconfiguration of the python code has it's own idea what to link in.

From make_simulator.log
{code}
....
temp.macosx-11.1-x86_64-3.9/src/checkdep_freetype2.o build/temp.macosx-11.1-x86_64-3.9/src/mplutils.o build/temp.macosx-11.1-x86_64-3.9/src/py_converters.o -lfreetype -lc++ -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.iossim.a -F/Users/olaf/Devel/oss/a-shell/cpython/Frameworks_iphonesimulator -F/Users/olaf/Devel/oss/a-shell/cpython/Frameworks_iphonesimulator
ld: building for iOS Simulator, but linking in dylib built for macOS, file '/usr/local/opt/freetype/lib/libfreetype.dylib' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang++' failed with exit code 1
{code}

[oflebbe]

I saw recommendation for other ports to compile in a VM ... maybe this is the way to go. I give up for now.
Thanks for your Support @holzschu !