freeze.py not working properly with OS X framework builds

[yjiangnan]

BPO	21502
Nosy	@ronaldoussoren, @ned-deily

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = None
created_at = <Date 2014-05-14.00:11:32.240>
labels = ['OS-mac', 'type-feature']
title = 'freeze.py not working properly with OS X framework builds'
updated_at = <Date 2015-02-07.15:33:25.994>
user = 'https://bugs.python.org/yjiangnan'

bugs.python.org fields:

activity = <Date 2015-02-07.15:33:25.994>
actor = 'ned.deily'
assignee = 'ned.deily'
closed = False
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2014-05-14.00:11:32.240>
creator = 'yjiangnan'
dependencies = []
files = []
hgrepos = []
issue_num = 21502
keywords = []
message_count = 2.0
messages = ['218493', '235528']
nosy_count = 2.0
nosy_names = ['ronaldoussoren', 'ned.deily']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21502'
versions = ['Python 3.4', 'Python 3.5']

Linked PRs

• gh-65701: document that freeze doesn't work with framework builds on macOS #113352
• [3.11] gh-65701: document that freeze doesn't work with framework builds on macOS (GH-113352) #113361
• [3.12] gh-65701: document that freeze doesn't work with framework builds on macOS (GH-113352) #113362

[yjiangnan]

Command:
python3.4 /Library/Frameworks/Python.framework/Versions/3.4/share/doc/python3.4/examples/Tools/freeze/freeze.py okcointrade.py

Error:
Reporting needed directory %s not found

Basically the problem is on lines 235~239: config should be config-3.4m, python%s on two lines should be python%sm

After I corrected these errors, the command works, asking me to run make, except a warning about "unknown modules remains".

So I run make, another error occurs:
gcc-4.2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c M_difflib.c
gcc-4.2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c M_ftplib.c
gcc-4.2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c M_pydoc.c
gcc-4.2 -Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.4/Python config.o frozen.o M___future__.o M___main__.o M__bootlocale.o M__collections_abc.o M__dummy_thread.o M__osx_support.o M__sitebuiltins.o M__strptime.o M__sysconfigdata.o M__threading_local.o M__weakrefset.o M_abc.o M_argparse.o M_ast.o M_base64.o M_bdb.o M_bisect.o M_bz2.o M_calendar.o M_cmd.o M_code.o M_codecs.o M_codeop.o M_collections.o M_contextlib.o M_copy.o M_copyreg.o M_datetime.o M_difflib.o M_dis.o M_distutils.o M_distutils__errors.o M_distutils__log.o M_distutils__spawn.o M_distutils__sysconfig.o M_distutils__text_file.o M_doctest.o M_dummy_threading.o M_email.o M_email___encoded_words.o M_email___header_value_parser.o M_email___parseaddr.o M_email___policybase.o M_email__base64mime.o M_email__charset.o M_email__contentmanager.o M_email__encoders.o M_email__errors.o M_email__feedparser.o M_email__generator.o M_email__header.o M_email__headerregistry.o M_email__iterators.o M_email__message.o M_email__parser.o M_email__policy.o M_email__quoprimime.o M_email__utils.o M_encodings.o M_encodings__aliases.o M_encodings__latin_1.o M_encodings__utf_8.o M_enum.o M_fnmatch.o M_ftplib.o M_functools.o M_genericpath.o M_getopt.o M_getpass.o M_gettext.o M_glob.o M_gzip.o M_hashlib.o M_heapq.o M_html.o M_html__entities.o M_http.o M_http__client.o M_http__cookiejar.o M_http__server.o M_imp.o M_importlib.o M_importlib___bootstrap.o M_importlib__machinery.o M_importlib__util.o M_inspect.o M_io.o M_json.o M_json__decoder.o M_json__encoder.o M_json__scanner.o M_keyword.o M_linecache.o M_locale.o M_logging.o M_lzma.o M_mimetypes.o M_ntpath.o M_nturl2path.o M_opcode.o M_operator.o M_optparse.o M_os.o M_pdb.o M_pkgutil.o M_platform.o M_plistlib.o M_posixpath.o M_pprint.o M_py_compile.o M_pydoc.o M_pydoc_data.o M_pydoc_data__topics.o M_quopri.o M_random.o M_re.o M_reprlib.o M_rlcompleter.o M_selectors.o M_shlex.o M_shutil.o M_site.o M_socket.o M_socketserver.o M_sre_compile.o M_sre_constants.o M_sre_parse.o M_ssl.o M_stat.o M_string.o M_struct.o M_subprocess.o M_sysconfig.o M_tarfile.o M_tempfile.o M_textwrap.o M_threading.o M_token.o M_tokenize.o M_traceback.o M_tty.o M_types.o M_unittest.o M_unittest__case.o M_unittest__loader.o M_unittest__main.o M_unittest__result.o M_unittest__runner.o M_unittest__signals.o M_unittest__suite.o M_unittest__util.o M_urllib.o M_urllib__error.o M_urllib__parse.o M_urllib__request.o M_urllib__response.o M_uu.o M_warnings.o M_weakref.o M_webbrowser.o M_xml.o M_xml__parsers.o M_xml__parsers__expat.o M_zipfile.o /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/config-3.4m/libpython3.4.a -ldl -framework CoreFoundation -o okcointrade
clang: error: no such file or directory: 'Python.framework/Versions/3.4/Python'
make: *** [okcointrade] Error 1

I guess that 'Python.framework/Versions/3.4/Python' using a relative location might be a problem, so I go into the Makefile, changed PYTHONFRAMEWORKDIR=Python.framework
into
PYTHONFRAMEWORKDIR=/Library/Frameworks/Python.framework

Now the above problem disappeared, but a new appeared:
gcc-4.2 -Wl,-stack_size,1000000 -framework CoreFoundation /Library/Frameworks/Python.framework/Versions/3.4/Python config.o frozen.o M___future__.o M___main__.o M__bootlocale.o M__collections_abc.o M__dummy_thread.o M__osx_support.o M__sitebuiltins.o M__strptime.o M__sysconfigdata.o M__threading_local.o M__weakrefset.o M_abc.o M_argparse.o M_ast.o M_base64.o M_bdb.o M_bisect.o M_bz2.o M_calendar.o M_cmd.o M_code.o M_codecs.o M_codeop.o M_collections.o M_contextlib.o M_copy.o M_copyreg.o M_datetime.o M_difflib.o M_dis.o M_distutils.o M_distutils__errors.o M_distutils__log.o M_distutils__spawn.o M_distutils__sysconfig.o M_distutils__text_file.o M_doctest.o M_dummy_threading.o M_email.o M_email___encoded_words.o M_email___header_value_parser.o M_email___parseaddr.o M_email___policybase.o M_email__base64mime.o M_email__charset.o M_email__contentmanager.o M_email__encoders.o M_email__errors.o M_email__feedparser.o M_email__generator.o M_email__header.o M_email__headerregistry.o M_email__iterators.o M_email__message.o M_email__parser.o M_email__policy.o M_email__quoprimime.o M_email__utils.o M_encodings.o M_encodings__aliases.o M_encodings__latin_1.o M_encodings__utf_8.o M_enum.o M_fnmatch.o M_ftplib.o M_functools.o M_genericpath.o M_getopt.o M_getpass.o M_gettext.o M_glob.o M_gzip.o M_hashlib.o M_heapq.o M_html.o M_html__entities.o M_http.o M_http__client.o M_http__cookiejar.o M_http__server.o M_imp.o M_importlib.o M_importlib___bootstrap.o M_importlib__machinery.o M_importlib__util.o M_inspect.o M_io.o M_json.o M_json__decoder.o M_json__encoder.o M_json__scanner.o M_keyword.o M_linecache.o M_locale.o M_logging.o M_lzma.o M_mimetypes.o M_ntpath.o M_nturl2path.o M_opcode.o M_operator.o M_optparse.o M_os.o M_pdb.o M_pkgutil.o M_platform.o M_plistlib.o M_posixpath.o M_pprint.o M_py_compile.o M_pydoc.o M_pydoc_data.o M_pydoc_data__topics.o M_quopri.o M_random.o M_re.o M_reprlib.o M_rlcompleter.o M_selectors.o M_shlex.o M_shutil.o M_site.o M_socket.o M_socketserver.o M_sre_compile.o M_sre_constants.o M_sre_parse.o M_ssl.o M_stat.o M_string.o M_struct.o M_subprocess.o M_sysconfig.o M_tarfile.o M_tempfile.o M_textwrap.o M_threading.o M_token.o M_tokenize.o M_traceback.o M_tty.o M_types.o M_unittest.o M_unittest__case.o M_unittest__loader.o M_unittest__main.o M_unittest__result.o M_unittest__runner.o M_unittest__signals.o M_unittest__suite.o M_unittest__util.o M_urllib.o M_urllib__error.o M_urllib__parse.o M_urllib__request.o M_urllib__response.o M_uu.o M_warnings.o M_weakref.o M_webbrowser.o M_xml.o M_xml__parsers.o M_xml__parsers__expat.o M_zipfile.o /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/config-3.4m/libpython3.4.a -ldl -framework CoreFoundation -o okcointrade
Undefined symbols for architecture x86_64:
"_PyInit__imp", referenced from:
__PyImport_Inittab in config.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [okcointrade] Error 1

Now it is out of my comprehension. What is wrong? How can I get freeze.py to work? I have a program doing very very heavy computation, so I really hope I can compile it into binary so that it could run much faster.

Thanks.

[ned-deily]

Sorry, I've finally gotten around to taking a longer look at this and it seems that freeze support is kind of a mess. First, it looks like some of the issues you ran into were fixed in 3.4.1 (changes associated with bpo-11824 and bpo-16047). Unfortunately, one of them introduced a new problem that I've documented in bpo-23405 (and which can be worked around by editing the freeze Makefile to use $(LIBRARY) instead of $(LDLIBRARY)). But there appear to be still more basic problems when trying to use freeze with OS X framework builds such as provided by the python.org installers for OS X.

1. freeze makes use of the LINKFORSHARED configuration variable when building the frozen executable. For OS X non-framework Python builds, it typically looks like this:
   LINKFORSHARED="-Wl,-stack_size,1000000 -framework CoreFoundation"
   For framework builds, it also includes the relative framework file path:
   LINKFORSHARED="-Wl,-stack_size,1000000 -framework CoreFoundation Python.framework/Versions/3.4/Python"
   The path is not useful as-is outside of the Python build itself and, by having a path here, the OS X framework configuration seems to be an exception to most other platforms, causing problems here with freeze and elsewhere (there have been other complaints about this usage). As noted, one can workaround this problem by editing the freeze-produced Makefile. But that leads to another problem.

2. Unlike for non-shared and shared Python builds, OS X Python framework builds do not include the Python library as a static archive file in the config-* directory (or anywhere else for that matter). Instead, symbolic links are created in the directory to the framework shared library.

shared build
$ ls -l ./lib/python3.4/config-3.4dm/lib*
-rw-r-----+ 1 nad pyd 9514552 Feb 7 16:57 libpython3.4dm.a

framework build
$ ls -l ./lib/python3.4/config-3.4dm/lib*
lrwxr-x--- 1 nad pyd 21 Feb 7 17:41 libpython3.4.a -> ../../../Python
lrwxr-x--- 1 nad pyd 21 Feb 7 17:41 libpython3.4.dylib -> ../../../Python
lrwxr-x--- 1 nad pyd 21 Feb 7 17:41 libpython3.4dm.a -> ../../../Python
lrwxr-x--- 1 nad pyd 21 Feb 7 17:41 libpython3.4dm.dylib -> ../../../Python

This difference causes problems when linking the frozen executable: because the framework build configuration presents a dynamic shared lib, the resulting executable has a runtime dependency on the shared lib:

% otool -L hello
hello:
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1152.0.0)
/Library/Frameworks/Python.framework/Versions/3.4/Python (compatibility version 3.4.0, current version 3.4.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

and the frozen executable cannot be run on another system which does not have a compatible version of Python installed. I don't know of an easy freeze build workaround for this. I think the correct solution for freeze is to always only provide the static library in the config* directory for all Python build configurations. But I'm not sure whether there are any compelling reasons to provide shared lib links as well for other users of the config* directory (e.g. for embedding Python); if so, the static and dynamic libs may have to have distinct names.

Ronald, do you have other thoughts on this?

At the moment, I think the safest solution for using freeze on OS X is to use a non-shared, non-framework build of Python, which probably means building Python 3.4 yourself.

[erlend-aasland]

Is this still relevant?

[FFY00]

@ned-deily would be able to see if this is still relevant?

[ned-deily]

@ned-deily would be able to see if this is still relevant?

I'll take a look.

[ronaldoussoren]

Ronald, do you have other thoughts on this?

Sorry, but I have never used freeze. But given the description in the readme I'd expect that freeze is not really useful with a framework (or shared library!) build because you'll end up with dependencies to the framework/shared library. This is worse on macOS than on other unix-y platforms because the load command for shared libraries contains an absolute path to the library (that is, a frozen script won't rely on libpython.so but on /path/to/libpython.so).

IMHO it is better to document that freeze is not compatible with framework builds, unless someone is willing to put in the work to fix issues (with test cases!) and document the limitations.

[erlend-aasland]

IMHO it is better to document that freeze is not compatible with framework builds [...]

+1

[ronaldoussoren]

The attached PR documents that framework builds aren't supported in the README file, and also bails out in freeze.py when it detects a framework build.

I looked into adding a test, but ended up not doing this because the test machinery tries to do a separate install in a temporary directory and that's problematic with framework builds as well (alternate prefix doesn't work, and is not something we should try to support because it is incompatible with platform standards)