Package request: python3-apt

[pschmitt]

Package description
This package provides APT python bindings required for Ansible. Without this, the apt: module cannot be used.

Link to home page and sources

1. Home page: https://packages.debian.org/search?keywords=python-apt
2. Source code: https://salsa.debian.org/apt-team/python-apt

Additional information
Have you compiled or tried to compile the package on device?

Yes. Without any patches the build fails with:

[...]
In file included from python/acquire.cc:26:
python/apt_pkgmodule.h:14:10: fatal error: 'apt-pkg/hashes.h' file not found
#include <apt-pkg/hashes.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.
error: command 'aarch64-linux-android-clang' failed with exit status 1

NOTE: The headers are provided by libapt-pkg-dev (not available on Termux either)

[Grimler91]

I've added a new package libapt-pkg, please give building on device another shot.
Note that you probably need to use the 1.4 branch of python-apt, as we have apt 1.4.9.

[ghost]

libapt-pkg is no longer separate package and is part of apt (splitting it makes apt unusable).

[pschmitt]

Thanks a lot guys!

I just tried what you suggested @Grimler91 and it works just fine (python-apt v1.4.0-beta).

ansible -i inventory.yml termux_remote -m apt -a "name=jq state=present"
termux_remote | SUCCESS => {
    "cache_update_time": 0,
    "cache_updated": false,
    "changed": false
}

Now I just need to figure out how to package a python library for Termux ;)
Should not be too hard since this is pretty much all that's required:

git clone --single-branch --branch 1.4.0_beta3 https://salsa.debian.org/apt-team/python-apt
cd python-apt
python setup.py install

Is there any package that is currently packaged similarly?

[pschmitt]

libapt-pkg is no longer separate package and is part of apt (splitting it makes apt unusable).

libapt-pkg still shows up when searching via apt. Is that intended?

[pschmitt]

asciinema seems fairly similar - with the exception that we need access to Python.h at build time.
I tried fiddling with LD_LIBRARY_PATH and PYTHONHOME - but that didn't get me anywhere. Is there something obvious that I am missing here?

See https://github.com/pschmitt/termux-packages/blob/python3-apt/packages/python3-apt/build.sh

[ghost]

libapt-pkg still shows up when searching via apt Is that intended?

I didn't removed previous versions of apt package, so assume that it is "intended".
Will remove it now.

[ghost]

I tried fiddling with LD_LIBRARY_PATH and PYTHONHOME - but that didn't get me anywhere

Building python modules (with exception for ones without native extensions) is unsupported by termux-packages.

Just python3.8 setup.py install --prefix=$TERMUX_PREFIX --force will not work in your build.sh script because it executes host python3.8 and not Termux's one which is cross-compiled and cannot be executed for host.

[Grimler91]

@xeffyr thanks for cleaning up my mess!

Running the host's python3.8 should be fine, that's what we do for asciinema and electrum. I'm more thinking that the Extension() call in setup.py needs to be patched to have the argument include_dirs = ['/data/data/com.termux/files/usr/include'], as done for example here: https://stackoverflow.com/a/10867041

[pschmitt]

@xeffyr thanks for cleaning up my mess!

Running the host's python3.8 should be fine, that's what we do for asciinema and electrum. I'm more thinking that the Extension() call in setup.py needs to be patched to have the argument include_dirs = ['/data/data/com.termux/files/usr/include'], as done for example here: https://stackoverflow.com/a/10867041

Thanks! That was exactly what I was looking for.
I actually managed to build a package this time, but it's a bit hackish (esp. the hardcoded /home/builder/.termux-build/apt/build/include path)

-> See https://github.com/pschmitt/termux-packages/blob/710ba64eef7883a9366e63bb31639f2addff85a0/packages/python3-apt/build.sh#L20

[ghost]

Isn't needed apt's include files available in $TERMUX_PREFIX/include ?
Latest apt package contains headers for apt-pkg, so you may try to use includes from $TERMUX_PREFIX instead of hardcoding builder path.

[pschmitt]

My bad. I didn't pull from origin/master beforehand. I removed the hardcoded path and updated my branch. Now I am able to build a package without any further fiddling. The only thing that I need to address now is that I end up with apt_inst.cpython-38-x86_64-linux-gnu.so and apt_pkg.cpython-38-x86_64-linux-gnu.so in the deb instead of arm libs - which obviously doesn't work. Is that what you were referring to @xeffyr with your comment about cross compilation?

[pschmitt]

If anyone is trying to get python-apt to work on termux the easiest (and only) way atm is:

pip install -e "git+https://salsa.debian.org/apt-team/python-apt.git@1.4.0_beta3#egg=python-apt"

[ghost]

Is that what you were referring to @xeffyr with your comment about cross compilation?

Yes. Python obtains all information about correct compiler and flags from sysconfig module. It seems to ignore $CC.

[pschmitt]

Hm. Judging by the output of the build command it actually does use aarch64-linux-android-clang:

./build-package.sh -i python3-apt
[...]
running build_ext                                    building 'apt_pkg' extension                         creating build/temp.linux-x86_64-3.8                 creating build/temp.linux-x86_64-3.8/python          aarch64-linux-android-clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fstack-protector-strong -Oz -I/data/data/com.termux/files/usr/include -fPIC -I/data/data/com.termux/files/usr/include -I/data/data/com.termux/files/usr/include/python3.8 -I/usr/include/python3.8 -c python/acquire.cc -o build/temp.linux-x86_64-3.8/python/a
cquire.o -std=c++11 -Wno-write-strings -DAPT_8_CLEANER_HEADERS -DAPT_9_CLEANER_HEADERS -DAPT_10_CLEANER_HEADERS
[...]

[pschmitt]

Update: the resulting .so files are ARM binaries:

file *.so                                                                                                                                              
apt_inst.cpython-38-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
apt_pkg.cpython-38-x86_64-linux-gnu.so:  ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped

Which sounds good, but even after creating symlinks manually, import apt fails:

cd /data/data/com.termux/files/usr/lib/python3.8/site-packages
ln -s apt_pkg.cpython-38-x86_64-linux-gnu.so apt_pkg.so
ln -s apt_inst.cpython-38-x86_64-linux-gnu.so apt_inst.so
python -c "import apt"                                                       
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ImportError: dlopen failed: cannot locate symbol "PyExc_ValueError" referenced by "/data/data/com.termux/files/usr/lib/python3.8/site-packages/apt_pkg.cpython-38-x86_64-linux-gnu.so"...

[cparker15]

If anyone is trying to get python-apt to work on termux the easiest (and only) way atm is:

pip install -e "git+https://salsa.debian.org/apt-team/python-apt.git@1.4.0_beta3#egg=python-apt"

Thanks for spelling this out, @pschmitt! I've adapted this command as an Ansible task:

- name: Ensure python-apt is installed on Termux hosts.
  # IMPORTANT: python and git packages must be installed manually on Termux hosts before this will succeed.
  ansible.builtin.pip:
    name: git+https://salsa.debian.org/apt-team/python-apt.git@{{ python_apt_version }}#egg=python-apt
  vars:
    python_apt_version: 2.2.1
  when: "'termux' in group_names"

. . . one three-hour-tour later . . .

---
# Run this on Termux host to test:
#   apt remove git python gcc-11 -y && apt autoremove -y && rm $PREFIX/etc/apt/sources.list.d/pointless.list && rm -rf $HOME/src && apt update && apt install python -y

- name: Ensure python-apt v{{ python_apt_version }} is installed on Termux hosts.
  when: "'termux' in group_names"
  tags: [ termux ]

  vars:
    python_apt_version: 2.2.1
    python_apt_url: git+https://salsa.debian.org/apt-team/python-apt.git@{{ python_apt_version }}#egg=python-apt

  block:

  - name: Check if python-apt is already installed.
    ansible.builtin.shell:
      cmd: 'python -c "import apt"'
    register: python_apt_is_installed
    changed_when: python_apt_is_installed.rc != 0
    failed_when: false

  - name: Install python-apt v{{ python_apt_version }} and dependencies.
    when: python_apt_is_installed.rc != 0
    vars:
      termux_home: /data/data/com.termux/files/home
      python_distutils_extra_version: 2.39
      python_distutils_extra_url: https://launchpad.net/python-distutils-extra/trunk/{{ python_distutils_extra_version }}/+download/python-distutils-extra-{{ python_distutils_extra_version }}.tar.gz
      python_distutils_extra_build_path: '{{ termux_home }}/python-distutils-extra-{{ python_distutils_extra_version }}'
      pointless_repo_setup_script_url: https://its-pointless.github.io/setup-pointless-repo.sh
      pointless_repo_setup_script_download_path: '{{ termux_home }}/setup-pointless-repo.sh'
      pointless_repo_list_path: /data/data/com.termux/files/usr/etc/apt/sources.list.d/pointless.list
    block:

    - name: Check if Git is already installed.
      ansible.builtin.shell:
        cmd: 'git --version'
      register: git_is_installed
      changed_when: git_is_installed.rc != 0
      failed_when: false
    
    - name: Install Git.
      when: git_is_installed.rc != 0
      block:

      - name: Install Git from official Termux repository.
        ansible.builtin.shell:
          cmd: 'apt install --upgrade git -y'
        register: git_install
        changed_when: '"git is already the newest version" not in git_install.stdout'

      - name: Check if Git installation succeeded.
        ansible.builtin.command:
          cmd: 'git --version'
        register: git_is_installed
        changed_when: false
        failed_when: git_is_installed.rc != 0

      rescue:
      - name: Git could not be installed.
        ansible.builtin.fail:
          msg: "WHOOPS!!! Git couldn't be installed! Sorry. :-("

    - name: Ensure latest versions of pip and sphinx are installed.
      ansible.builtin.pip:
        name:
        - pip
        - sphinx
        state: latest

    - name: Check if python-distutils-extra v{{ python_distutils_extra_version }} is already installed.
      ansible.builtin.shell:
        cmd: "pip list | grep python-distutils-extra | grep {{ python_distutils_extra_version }}"
      register: python_distutils_extra_is_installed
      changed_when: not python_distutils_extra_is_installed.stdout.startswith('python-distutils-extra')
      failed_when: false

    - name: Install python-distutils-extra v{{ python_distutils_extra_version }}.
      when: not python_distutils_extra_is_installed.stdout.startswith('python-distutils-extra')
      block:

      - name: Download python-distutils-extra v{{ python_distutils_extra_version }}.
        ansible.builtin.unarchive:
          src: "{{ python_distutils_extra_url }}"
          dest: "{{ termux_home }}"
          remote_src: yes
          mode: 0755
          creates: '{{ python_distutils_extra_build_path }}/setup.py'

      - name: Build python-distutils-extra v{{ python_distutils_extra_version }}.
        ansible.builtin.shell:
          chdir: "{{ python_distutils_extra_build_path }}"
          cmd: ./setup.py build

      - name: Install python-distutils-extra v{{ python_distutils_extra_version }}.
        ansible.builtin.shell:
          chdir: "{{ python_distutils_extra_build_path }}"
          cmd: ./setup.py install

      - name: Cleanup python-distutils-extra v{{ python_distutils_extra_version }} build files post-install.
        ansible.builtin.file:
          path: "{{ python_distutils_extra_build_path }}"
          state: absent

      - name: Check if python-distutils-extra v{{ python_distutils_extra_version }} installation succeeded.
        ansible.builtin.shell:
          cmd: "pip list | grep python-distutils-extra | grep {{ python_distutils_extra_version }}"
        register: python_distutils_extra_is_installed
        changed_when: false
        failed_when: not python_distutils_extra_is_installed.stdout.startswith('python-distutils-extra')

      rescue:
      - name: python-distutils-extra v{{ python_distutils_extra_version }} could not be installed.
        ansible.builtin.fail:
          msg: "WHOOPS!!! python-distutils-extra v{{ python_distutils_extra_version }} couldn't be installed! Sorry. :-("

    - name: Check if 'pointless' Termux repository already exists.
      ansible.builtin.stat:
        path: "{{ pointless_repo_list_path }}"
      register: pointless_repo_exists
      changed_when: not pointless_repo_exists.stat.exists

    - name: Install 'pointless' Termux repository.
      when: not pointless_repo_exists.stat.exists
      block:

      - name: Download 'pointless' Termux repository setup script.
        ansible.builtin.get_url:
          url: "{{ pointless_repo_setup_script_url }}"
          dest: "{{ pointless_repo_setup_script_download_path }}"
          mode: 0755
        register: pointless_repo_setup_script_downloaded

      - name: Run 'pointless' Termux repository setup script.
        ansible.builtin.shell:
          cmd: "{{ pointless_repo_setup_script_download_path }}"
        when: pointless_repo_setup_script_downloaded.changed

      - name: Cleanup 'pointless' Termux repository setup script post-install.
        ansible.builtin.file:
          path: "{{ pointless_repo_setup_script_download_path }}"
          state: absent
        when: pointless_repo_setup_script_downloaded.changed

      - name: Check if 'pointless' Termux repository installation succeeded.
        ansible.builtin.stat:
          path: "{{ pointless_repo_list_path }}"
        register: pointless_repo_exists
        changed_when: false
        failed_when: not pointless_repo_exists.stat.exists

      rescue:
      - name: The 'pointless' Termux repository could not be installed.
        ansible.builtin.fail:
          msg: "WHOOPS!!! The 'pointless' Termux repository couldn't be installed! Sorry. :-("

    - name: Install GCC 11 from 'pointless' Termux repository.
      ansible.builtin.shell:
        cmd: apt install --upgrade gcc-11 -y
      register: gcc_install
      changed_when: '"gcc-11 is already the newest version" not in gcc_install.stdout'

    - name: Install python-apt v{{ python_apt_version }}.
      ansible.builtin.shell:
        cmd: 'pip install -e "{{ python_apt_url }}"'
      register: python_apt_install
      failed_when: '"Successfully installed python-apt" not in python_apt_install.stdout'

    - name: Check if python-apt installation succeeded.
      ansible.builtin.command:
        cmd: 'python -c "import apt"'
      register: python_apt_is_installed
      changed_when: false
      failed_when: python_apt_is_installed.rc != 0

[xtkoba]

ImportError: dlopen failed: cannot locate symbol "PyExc_ValueError" referenced by "/data/data/com.termux/files/usr/lib/python3.8/site-packages/apt_pkg.cpython-38-x86_64-linux-gnu.so"...

Just hitting android/ndk#201. Needs something like:

export LDFLAGS+=" -L${PREFIX}/lib -lpython3.8"

(adapt to the Python version you have).