Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] ELF load command address/offset not properly aligned #1048

Closed
mcauto opened this issue May 10, 2021 · 16 comments
Closed

[Bug] ELF load command address/offset not properly aligned #1048

mcauto opened this issue May 10, 2021 · 16 comments

Comments

@mcauto
Copy link

mcauto commented May 10, 2021

I install pacakage after bdist_rpm.

example project: https://github.com/mcauto/cx-freeze-bdist-rpm-example

# host
$ git clone https://github.com/mcauto/cx-freeze-bdist-rpm-example
$ cd cx-freeze-bdist-rpm-example
$ docker-compose up --build -d
$ docker exec -it build-test_app_1 bash
# in docker
$ python setup.py bdist_rpm
$ rpm -ivh dist/*.rpm

Then occur error message below.
because of paramiko

sh-4.2# build_test
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cx_Freeze/initscripts/__startup__.py", line 81, in run
    module.run()
  File "/usr/local/lib/python3.9/site-packages/cx_Freeze/initscripts/Console.py", line 36, in run
    exec(code, m.__dict__)
  File "build_test/__main__.py", line 1, in <module>
    from build_test.hello import hello
  File "<frozen zipimport>", line 259, in load_module
  File "/code/build/bdist.linux-x86_64/rpm/BUILD/build_test-0.0.1/build_test/hello.py", line 1, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/paramiko/__init__.py", line 22, in <module>
    from paramiko.transport import SecurityOptions, Transport
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/paramiko/transport.py", line 89, in <module>
    from paramiko.dsskey import DSSKey
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/paramiko/dsskey.py", line 37, in <module>
    from paramiko.pkey import PKey
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/paramiko/pkey.py", line 31, in <module>
    import bcrypt
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/bcrypt/__init__.py", line 25, in <module>
    from . import _bcrypt  # type: ignore
ImportError: /usr/local/lib/build_test-0.0.1/lib/_cffi_backend.cpython-39-x86_64-linux-gnu.so: ELF load command address/offset not properly aligned
@marcelotduarte
Copy link
Owner

Desktop (please complete the following information):

  • Platform information (e.g. Ubuntu Linux 16.04):
  • OS architecture (e.g. amd64):
  • cx_Freeze version [e.g. 6.6]:
  • Python version [e.g. 3.9]:

@mcauto
Copy link
Author

mcauto commented May 11, 2021

Desktop (please complete the following information):

Here !

  • Platform information (e.g. Ubuntu Linux 16.04): centos 7 (with docker)
  • OS architecture (e.g. amd64): x86_64
  • cx_Freeze version [e.g. 6.6]: 6.6
  • Python version [e.g. 3.9]: 3.9

@marcelotduarte
Copy link
Owner

I looked at your example, over the top, I don't have time to test it now. However, I see that you use a version of python compiled from sources, so I believe that some configuration options are missing (line 15 of the Dockerfile). Use:
configure --enable-shared
Maybe you need to add --enable-ipv6 if use some networking
And the flag:
LDFLAGS_NODIST="-Wl,-rpath,/usr/local/lib"
If you have some other error.
Try it out!

@marcelotduarte
Copy link
Owner

Do you have success?

@mcauto
Copy link
Author

mcauto commented May 20, 2021

I cant solve it..

I try it like below.
however, same error occurs.

...
RUN tar xJvf Python-3.9.5.tar.xz \
    && cd Python-3.9.5 \
    && ./configure --enable-shared --enable-ipv6
RUN cd Python-3.9.5 \
    && LDFLAGS_NODIST="-Wl,-rpath,/usr/local/lib" make \
    && make altinstall
...
sh-4.2# build_test
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cx_Freeze/initscripts/__startup__.py", line 81, in run
    module.run()
  File "/usr/local/lib/python3.9/site-packages/cx_Freeze/initscripts/Console.py", line 36, in run
    exec(code, m.__dict__)
  File "build_test/__main__.py", line 1, in <module>
    from build_test.hello import hello
  File "<frozen zipimport>", line 259, in load_module
  File "/code/build/bdist.linux-x86_64/rpm/BUILD/build_test-0.0.1/build_test/hello.py", line 1, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/paramiko/__init__.py", line 22, in <module>
    from paramiko.transport import SecurityOptions, Transport
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/paramiko/transport.py", line 89, in <module>
    from paramiko.dsskey import DSSKey
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/paramiko/dsskey.py", line 37, in <module>
    from paramiko.pkey import PKey
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/paramiko/pkey.py", line 31, in <module>
    import bcrypt
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/bcrypt/__init__.py", line 25, in <module>
    from . import _bcrypt  # type: ignore
ImportError: /usr/local/lib/build_test-0.0.1/lib/_cffi_backend.cpython-39-x86_64-linux-gnu.so: ELF load command address/offset not properly aligned

@marcelotduarte
Copy link
Owner

...
&& ./configure --enable-shared --enable-ipv6 LDFLAGS_NODIST="-Wl,-rpath,/usr/local/lib" \
&& make
...

@marcelotduarte
Copy link
Owner

marcelotduarte commented May 29, 2021

Do you get this project running in this image? There are some issues to run a py39 in centos7.
In manylinux they solve them.
But, the shared library isn't included.
So, if you use: FROM quay.io/pypa/manylinux2014_x86_64
You'll get an error.

I had preliminary work with manylinux2010 (centos6) but you can adapt to use manylinux2014 (centos7).
See https://github.com/marcelotduarte/cx_Freeze/tree/develop/ci
https://github.com/marcelotduarte/cx_Freeze/blob/develop/ci/build-manylinux2010.sh
https://github.com/marcelotduarte/cx_Freeze/blob/develop/ci/build-manylinux-create.sh

@mcauto
Copy link
Author

mcauto commented May 29, 2021

I use CentOS 7(+python3.9) on-premise server in production.

I try it, then i get an error.
Could you push your docker image(manylinux-freeze) on docker hub?

# https://github.com/pypa/manylinux
FROM quay.io/pypa/manylinux2014_x86_64
COPY requirements.txt requirements.txt
python3.9 -m install -r requirements.txt
Installing collected packages: cx-freeze
    Running setup.py install for cx-freeze ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_bkrfu6z/cx-freeze_c851ee3ebfbb4cabad3e02bae3465dcb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_bkrfu6z/cx-freeze_c851ee3ebfbb4cabad3e02bae3465dcb/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-7duxfrih/install-record.txt --single-version-externally-managed --compile --install-headers /opt/_internal/cpython-3.9.5/include/python3.9/cx-freeze
         cwd: /tmp/pip-install-_bkrfu6z/cx-freeze_c851ee3ebfbb4cabad3e02bae3465dcb/
    Complete output (40 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/finder.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/cli.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/macdist.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/__main__.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/setupwriter.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/common.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/windist.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/darwintools.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/dist.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/freezer.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/__init__.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/executable.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/winversioninfo.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/winmsvcr.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/exception.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/patchelf.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/module.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    copying cx_Freeze/hooks.py -> build/lib.linux-x86_64-3.9/cx_Freeze
    creating build/lib.linux-x86_64-3.9/cx_Freeze/initscripts
    copying cx_Freeze/initscripts/ConsoleSetLibPath.py -> build/lib.linux-x86_64-3.9/cx_Freeze/initscripts
    copying cx_Freeze/initscripts/SharedLib.py -> build/lib.linux-x86_64-3.9/cx_Freeze/initscripts
    copying cx_Freeze/initscripts/__startup__.py -> build/lib.linux-x86_64-3.9/cx_Freeze/initscripts
    copying cx_Freeze/initscripts/SharedLibSource.py -> build/lib.linux-x86_64-3.9/cx_Freeze/initscripts
    copying cx_Freeze/initscripts/Console.py -> build/lib.linux-x86_64-3.9/cx_Freeze/initscripts
    running build_ext
    creating build/temp.linux-x86_64-3.9
    creating build/temp.linux-x86_64-3.9/source
    creating build/temp.linux-x86_64-3.9/source/bases
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/_internal/cpython-3.9.5/include/python3.9 -c source/bases/Console.c -o build/temp.linux-x86_64-3.9/source/bases/Console.o
    creating build/lib.linux-x86_64-3.9/cx_Freeze/bases
    gcc -pthread build/temp.linux-x86_64-3.9/source/bases/Console.o -L/opt/_internal/cpython-3.9.5/lib/python3.9/config-3.9-x86_64-linux-gnu -lpython3.9 -o build/lib.linux-x86_64-3.9/cx_Freeze/bases/Console -Xlinker -export-dynamic -lcrypt -lpthread -ldl -lutil -lm -lm -s -Wl,-rpath,$ORIGIN/lib -Wl,-rpath,$ORIGIN/../lib
    /opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: cannot find -lpython3.9
    collect2: error: ld returned 1 exit status
    error: command '/opt/rh/devtoolset-9/root/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_bkrfu6z/cx-freeze_c851ee3ebfbb4cabad3e02bae3465dcb/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_bkrfu6z/cx-freeze_c851ee3ebfbb4cabad3e02bae3465dcb/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-7duxfrih/install-record.txt --single-version-externally-managed --compile --install-headers /opt/_internal/cpython-3.9.5/include/python3.9/cx-freeze Check the logs for full command output.

@marcelotduarte
Copy link
Owner

I try it, then i get an error.

Like I said.

Could you push your docker image(manylinux-freeze) on docker hub?

The reserved space for open source is github registry for instance is 500MB. The image is about 900MB.

But you can produce it.

Install docker with buildx
(
if your system do not have it, you can use:
curl -fsSL https://get.docker.com/ | sh
sudo usermod -aG docker $USER
and restart
)

Clone manylinux
Clone develop branch of cx_Freeze: git clone -b develop https://github.com/marcelotduarte/cx_Freeze.git
create a venv and go to the cx_Freeze directory
edit ci/build-manylinux2010.sh to comment the lines
./build-manylinux-run.sh
./build-manylinux-tests.sh
run:
./ci/build-manylinux2010.sh

Dockerfile:

FROM manylinux2010_x86_64

RUN yum install -y rpm-build

RUN ln -s /usr/local/bin/python3.9 /usr/local/bin/python3

RUN python3 -m pip install --upgrade git+https://github.com/marcelotduarte/cx_Freeze.git

WORKDIR /code
COPY requirements.txt requirements.txt
RUN python3 -m pip install -r requirements.txt

# cython import
#RUN pip install -U \
#    git+https://github.com/marcelotduarte/cx_Freeze.git@main 

COPY setup.py setup.py
COPY build_test build_test
RUN set -ex \
    && python3 setup.py build_exe --silent bdist_rpm
RUN rpm -ivh dist/*.rpm

CMD ["build_test"]

@mcauto
Copy link
Author

mcauto commented Jun 1, 2021

Thank you kindful explain!

but I get an error.
when I install build_test.rpm (build on manylinux2010_x86_64) on centos 7

# centos 7
drwxr-xr-x  2 root root     4096 Jun  1 01:56 .
drwxr-xr-x 12 root root     4096 Nov 13  2020 ..
-rw-r--r--  1 root root     3469 Jun  1 01:56 build_test-0.0.1-1.src.rpm
-rw-r--r--  1 root root 10330108 Jun  1 01:56 build_test-0.0.1-1.x86_64.rpm
-rw-r--r--  1 root root     1284 Jun  1 01:56 build_test-0.0.1.tar.gz
-rw-r--r--  1 root root  1461436 Jun  1 01:56 build_test-debuginfo-0.0.1-1.x86_64.rpm
[root@3d94eacf9197 src]# rpm -ivh *.rpm
error: Failed dependencies:
        libffi.so.5()(64bit) is needed by build_test-0.0.1-1.x86_64
        liblzma.so.0()(64bit) is needed by build_test-0.0.1-1.x86_64
Updating / installing...
   1:build_test-0.0.1-1               ################################# [100%]
[root@3d94eacf9197 src]#
# manylinux2010_x86_64
$ exec /opt/_internal/cpython-3.9.5/bin/build_test
ImportError: /opt/_internal/cpython-3.9.5/lib/build_test-0.0.1/lib/_cffi_backend.cpython-39-x86_64-linux-gnu.so: ELF load command address/offset not properly aligned

@marcelotduarte
Copy link
Owner

I get it running with more changes:
in setup.py add "bin_path_includes": "/usr/lib64", to options build_exe
in requirements change to cffi==1.13.2 (Why? https://foss.heptapod.net/pypy/cffi/-/issues/432)
Dockerfile:

# docker build -t build-test-rpm
# docker run --rm build-test-rpm
FROM manylinux2010_x86_64 as builder

RUN yum install -y rpm-build libffi-devel

RUN ln -s /usr/local/bin/python3.9 /usr/local/bin/python3

WORKDIR /code
COPY requirements.txt requirements.txt
RUN python3 -m pip install -r requirements.txt

COPY setup.py setup.py
COPY build_test build_test
RUN set -ex \
    && python3 setup.py bdist_rpm

FROM centos:7
COPY --from=builder /code/dist/*.rpm /dist/
RUN rpm -ivh dist/*.rpm
USER nobody:nobody
ENTRYPOINT ["/opt/_internal/cpython-3.9.5/bin/build_test"]

@marcelotduarte
Copy link
Owner

Deo, I get success without changing the setup and requirements.
There are four issues:

  1. --enable-shared (using my modified manylinx solves this)
  2. corruption by patchelf in the libffi dependency
  3. corruption by strip in rpmbuild (using strip) in the cffi extension
    Check the PR for more details.
  4. failed dependencies - use bdist_rpm --no-autoreq

Dockerfile:

# docker build -t build-test-rpm .
# docker run --rm build-test-rpm
FROM manylinux2010_x86_64 as builder

RUN yum install -y rpm-build
# libffi-devel openssl-devel

RUN ln -s /opt/_internal/cpython-3.9.5/bin/python3.9 /usr/local/bin/python3
RUN ln -s /opt/_internal/cpython-3.9.5/bin/pip3.9 /usr/local/bin/pip

WORKDIR /code
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
RUN pip uninstall -y cx-freeze && \
    pip install -U git+https://github.com/marcelotduarte/cx_Freeze.git@develop

COPY setup.py setup.py
COPY build_test build_test
RUN set -ex && \
    python3 setup.py bdist_rpm --no-autoreq
RUN ls -l build/bdist*/rpm/*/* || true
RUN ldd /opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/_cffi_backend.cpython-39-x86_64-linux-gnu.so
RUN patchelf --print-soname --print-rpath --print-needed --debug /opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/_cffi_backend.cpython-39-x86_64-linux-gnu.so

FROM centos:7
COPY --from=builder /code/dist/* /dist/
COPY --from=builder /usr/local/bin/patchelf /usr/local/bin/patchelf
RUN rpm -ivh dist/build_test-0.0.1-1.x86_64.rpm
RUN ls -l -h /opt/_internal/cpython-3.9.5/lib/build_test-0.0.1
RUN ldd /opt/_internal/cpython-3.9.5/lib/build_test-0.0.1/lib/*.so* | grep -i -B 1 "not a dynamic executable" || true
RUN patchelf --print-soname --print-rpath --print-needed --debug /opt/_internal/cpython-3.9.5/lib/build_test-0.0.1/lib/_cffi_backend.cpython-39-x86_64-linux-gnu.so
USER nobody:nobody
RUN /opt/_internal/cpython-3.9.5/lib/build_test-0.0.1/build_test
CMD ["/opt/_internal/cpython-3.9.5/lib/build_test-0.0.1/build_test"]

@mcauto
Copy link
Author

mcauto commented Jun 10, 2021

Awesome! It's works!

Unfortunately, I was faced with other problems...
When I add ldap3 dependency.
Occur error blow.😨
Could you help me one more..? plz

pip install ldap3
# build_test/hello.py
import paramiko
import ldap3
def hello():
    print("hello")
ImportError libffi.so.5: cannot open shared object file: No such file or directory
log
 => ERROR [stage-1 8/8] RUN /opt/_internal/cpython-3.9.5/lib/build_test-0.0.1/build_test                                                                                                                                             1.0s
------
 > [stage-1 8/8] RUN /opt/_internal/cpython-3.9.5/lib/build_test-0.0.1/build_test:
#27 0.922 Traceback (most recent call last):
#27 0.922   File "/opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/cx_Freeze/initscripts/__startup__.py", line 81, in run
#27 0.924   File "/opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/cx_Freeze/initscripts/Console.py", line 36, in run
#27 0.924   File "build_test/__main__.py", line 1, in <module>
#27 0.924   File "<frozen zipimport>", line 259, in load_module
#27 0.924   File "/code/build/bdist.linux-x86_64/rpm/BUILD/build_test-0.0.1/build_test/hello.py", line 2, in <module>
#27 0.925   File "<frozen zipimport>", line 259, in load_module
#27 0.925   File "/opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/ldap3/__init__.py", line 141, in <module>
#27 0.925   File "<frozen zipimport>", line 259, in load_module
#27 0.925   File "/opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/ldap3/core/connection.py", line 38, in <module>
#27 0.925   File "<frozen zipimport>", line 259, in load_module
#27 0.925   File "/opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/ldap3/extend/__init__.py", line 29, in <module>
#27 0.925   File "<frozen zipimport>", line 259, in load_module
#27 0.925   File "/opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/ldap3/extend/microsoft/dirSync.py", line 27, in <module>
#27 0.925   File "<frozen zipimport>", line 259, in load_module
#27 0.925   File "/opt/_internal/cpython-3.9.5/lib/python3.9/site-packages/ldap3/protocol/microsoft.py", line 26, in <module>
#27 0.925   File "<frozen zipimport>", line 259, in load_module
#27 0.925   File "/opt/_internal/cpython-3.9.5/lib/python3.9/ctypes/__init__.py", line 8, in <module>
#27 0.925 ImportError: libffi.so.5: cannot open shared object file: No such file or directory

@marcelotduarte
Copy link
Owner

In this specific case, use "bin_includes": "libffi.so.5"

    options={
        "build_exe":{
            "zip_include_packages": ["*"],
            "zip_exclude_packages": [],
            "includes": [],
            "silent": True,
            "bin_includes": "libffi.so.5",
        }
    },

@mcauto
Copy link
Author

mcauto commented Jun 10, 2021

Thank you. It's works!
close issue.

@mcauto mcauto closed this as completed Jun 10, 2021
@marcelotduarte
Copy link
Owner

cx_Freeze 6.7 has just been released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants