Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to install python-apt package because of DistUtilsExtra module not found #1363

Closed
2 tasks done
Kocal opened this issue Sep 7, 2019 · 9 comments
Closed
2 tasks done

Comments

@Kocal
Copy link

Kocal commented Sep 7, 2019

  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have searched the documentation and believe that my question is not covered.

Question

Hi!

Some info before asking the question:

  • Poetry 1.0.0b1
  • Poetry setup a virtualenv with Python 3.7

I'm trying to install python-apt package (for using apt module in Ansible), but I'm facing this error:

➜  System git:(feat/dconf) ✗ poetry add python-apt                       
Using version ^0.7.8 for python-apt

Updating dependencies
Resolving dependencies... (0.1s)


Package operations: 1 install, 0 updates, 0 removals

  - Installing python-apt (0.7.8)

[EnvCommandError]
Command ['/home/kocal/.cache/pypoetry/virtualenvs/system-H90qh8t--py3.7/bin/python', '-m', 'pip', 'install', '--no-deps', 'python-apt==0.7.8'] errored with the following return code 1, and output: 
Collecting python-apt==0.7.8
  Using cached https://files.pythonhosted.org/packages/e5/ff/63bb64a103eda6f13364381e983c0121eeacc337a4421d6005ff7dd76741/python-apt-0.7.8.tar.bz2
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-hcb_ok5s/python-apt/setup.py", line 6, in <module>
        from DistUtilsExtra.command import *
    ModuleNotFoundError: No module named 'DistUtilsExtra'

Indeed, I don't see any distutils library in my virtualenv libraries:

➜  System git:(feat/dconf) ✗ l ~/.cache/pypoetry/virtualenvs/system-H90qh8t--py3.7/lib/python3.7/site-packages
total 88K
drwxr-xr-x 21 kocal kocal 4,0K sept.  7 07:37 .
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:36 ..
drwxr-xr-x 18 kocal kocal 4,0K sept.  7 07:36 ansible
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 ansible-2.7.5.egg-info
-rw-r--r--  1 kocal kocal  126 sept.  7 07:36 easy_install.py
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:37 jinja2
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 Jinja2-2.10.dist-info
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:37 jmespath
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 jmespath-0.9.3.dist-info
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:36 markupsafe
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 MarkupSafe-1.1.0.dist-info
drwxr-xr-x  5 kocal kocal 4,0K sept.  7 07:36 pip
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 pip-18.1.dist-info
drwxr-xr-x  5 kocal kocal 4,0K sept.  7 07:36 pkg_resources
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 pkg_resources-0.0.0.dist-info
drwxr-xr-x  4 kocal kocal 4,0K sept.  7 07:37 psutil
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 psutil-5.6.3.egg-info
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 __pycache__
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:37 PyYAML-5.1.1.egg-info
drwxr-xr-x  6 kocal kocal 4,0K sept.  7 07:36 setuptools
drwxr-xr-x  2 kocal kocal 4,0K sept.  7 07:36 setuptools-40.8.0.dist-info
drwxr-xr-x  3 kocal kocal 4,0K sept.  7 07:37 yaml

But I don't find how to install distutils with Poetry.
It seems that distutils is only installable with system's package manager: sudo apt-get install python3-distutils.

I know that I've already ran this command, so distutils is installed for the global Python interpreter:

➜  System git:(feat/dconf) ✗ l /usr/lib/python3.7/distutils
total 500K
drwxr-xr-x  4 root root 4,0K juin   1 06:27 .
drwxr-xr-x 31 root root  20K juil. 12 06:33 ..
-rw-r--r--  1 root root 8,4K mars  26 08:58 archive_util.py
-rw-r--r--  1 root root  15K mars  26 08:58 bcppcompiler.py
-rw-r--r--  1 root root  47K mars  26 08:58 ccompiler.py
-rw-r--r--  1 root root  18K mars  26 08:58 cmd.py
drwxr-xr-x  3 root root 4,0K juin   1 06:27 command
-rw-r--r--  1 root root 4,8K mars  26 08:58 config.py
-rw-r--r--  1 root root 8,7K mars  26 08:58 core.py
-rw-r--r--  1 root root  17K mars  26 08:58 cygwinccompiler.py
-rw-r--r--  1 root root  139 mars  26 08:58 debug.py
-rw-r--r--  1 root root 3,5K mars  26 08:58 dep_util.py
-rw-r--r--  1 root root 8,2K mars  26 08:58 dir_util.py
-rw-r--r--  1 root root  50K mars  26 08:58 dist.py
-rw-r--r--  1 root root 3,5K mars  26 08:58 errors.py
-rw-r--r--  1 root root  11K mars  26 08:58 extension.py
-rw-r--r--  1 root root  18K mars  26 08:58 fancy_getopt.py
-rw-r--r--  1 root root  13K mars  26 08:58 filelist.py
-rw-r--r--  1 root root 8,0K mars  26 08:58 file_util.py
-rw-r--r--  1 root root  236 avril  3 07:39 __init__.py
-rw-r--r--  1 root root 2,0K mars  26 08:58 log.py
-rw-r--r--  1 root root  30K mars  26 08:58 msvc9compiler.py
-rw-r--r--  1 root root  22K mars  26 08:58 _msvccompiler.py
-rw-r--r--  1 root root  24K mars  26 08:58 msvccompiler.py
drwxr-xr-x  2 root root 4,0K juin   1 06:27 __pycache__
-rw-r--r--  1 root root  295 mars  26 08:58 README
-rw-r--r--  1 root root 7,3K mars  26 08:58 spawn.py
-rw-r--r--  1 root root  22K mars  26 08:58 sysconfig.py
-rw-r--r--  1 root root  13K mars  26 08:58 text_file.py
-rw-r--r--  1 root root  15K mars  26 08:58 unixccompiler.py
-rw-r--r--  1 root root  20K mars  26 08:58 util.py
-rw-r--r--  1 root root 5,1K mars  26 08:58 versionpredicate.py
-rw-r--r--  1 root root  13K avril  3 07:39 version.py

but I can't use it through Poetry.

Did anyone already encountered the issue? 😕

Thanks!

@rdonkin
Copy link

rdonkin commented Sep 20, 2019

I had a similar issue, starting with this error from Ansible when using a non-APT Python 3.7 in a virtualenv:

msg: Could not import python modules: apt, apt_pkg. Please install python-apt package.

Trying to install python-apt led to the same problem that you had with the error about DistUtilsExtra, but installing python-apt from PyPA is not the right solution, as it's very old, and distutils is a red herring as well.

Instead, here's what did work for Ubuntu 18.04, by installing the APT package python3-apt into the virtualenv in a rather hacky way.

I'm not using poetry for this but I am using a virtualenv and Python 3.7 (to get ansible-vault to decrypt properly), so I think this solution will work along with poetry. (BTW I like poetry but have other priorities right now.)

This is based on the very helpful solution by @acardak - that issue is locked so I thought I would comment here. I had to adjust that solution a little (version numbers in filenames) so it worked on fully updated Ubuntu 18.04.

#!/bin/bash
# Before starting, do `apt install python3.7` and 
# ensure `python3` is a symlink to that binary, 
# so that `python3 --version` reports 3.7.

# Installs python3.7 binary into virtualenv as python3
python3 -m virtualenv ~/venv-py37           
source ~/venv-py37/bin/activate

# Install python3-apt package into virtualenv, based on 
# https://github.com/ansible/ansible/issues/14468#issuecomment-459630445
# This package must be installed carefully so it works inside virtualenv 
# with Python 3.7, not the system default Python 3.6

# Download and extract python3-apt files into /tmp/python3-apt
pushd /tmp
apt-get download python3-apt
dpkg -x python3-apt_1.6.4_amd64.deb python3-apt
popd

# Install python3-apt files into virtualenv
cp -r /tmp/python3-apt/usr/lib/python3/dist-packages/* ~/venv-py37/lib/python3.7/site-packages/

# Rename shared libs inside the virtualenv packages dir
pushd ~/venv-py37/lib/python3.7/site-packages/
mv apt_pkg.cpython-36m-x86_64-linux-gnu.so apt_pkg.so
mv apt_inst.cpython-36m-x86_64-linux-gnu.so apt_inst.so
popd

@Kocal
Copy link
Author

Kocal commented Sep 20, 2019

Hum interesting, I will give it a try when I will be back home.
Thanks! :)

@eff917
Copy link

eff917 commented Nov 20, 2019

The problem with python-apt is that the package is not updated on pypi. I was able to install it from this repo: https://salsa.debian.org/apt-team/python-apt/

@finswimmer
Copy link
Member

finswimmer commented Nov 20, 2019

I close this issue as it seems to be an upstream problem and not by poetry.

If someone not agree with it, please let me know.

@refulgent-de
Copy link

refulgent-de commented Aug 16, 2020

Your solution helped me alot, @rdonkin. To show my gratitude I allowed myself to port your solution into an ansible script : )

- name: Install python3-apt 
  when:
    - ansible_facts['distribution'] == 'Ubuntu'
    - ansible_facts['distribution_major_version'] == '18'
  block:

    - file:
        path: "{{ virtualenvDirectory }}/src"
        state: directory
        owner: "{{ systemUser }}"
        group: "{{ systemUser }}"
        mode: '0750'

    - name: Download python3-apt package
      shell: "apt-get download python3-apt"
      args:
        chdir: "{{ virtualenvDirectory }}/src"
        warn: false
        
    - name: Get filename of downloaded package
      find:
        paths: "{{ virtualenvDirectory }}/src"
        file_type: file
        patterns: python3-apt*
      register: findResult
  
    - fail:
        msg: Downloaded package was not found
      when: findResult is undefined or findResult.files|length == 0

    - set_fact: 
        python3AptPackageFileName: "{{ findResult.files[0].path|regex_replace('^.*/(.*)$','\\1') }}"
      when: findResult is defined

    - name: Extract package
      shell: dpkg -x {{ python3AptPackageFileName }} python3-apt
      args:
        chdir: "{{ virtualenvDirectory }}/src"

    - name: Get list of sharable objects
      find:
        paths: "{{ virtualenvDirectory }}/src/python3-apt/usr/lib/python3/dist-packages" # Path should stay stable throughout python 3.x I guess
        file_type: file
        patterns: '*.so'
      register: findResult
    - set_fact:
        python3AptShareableObjectsFilenames: "{{ python3AptShareableObjectsFilenames|default([]) + [item.path|regex_replace('^.*/(.*)$','\\1')] }}"
      with_items: "{{ findResult.files }}"

    - name: Copy package into virtualenv lib directory
      copy:
        src: "{{ virtualenvDirectory }}/src/python3-apt/usr/lib/python3/dist-packages/"
        dest: "{{ virtualenvDirectory }}/lib/{{ altInterpreterFileName }}/site-packages/"
        remote_src: yes
        owner: "{{ systemUser }}"
        group: "{{ systemUser }}"
        mode: '0750'

    - name: Rename shareable objects
      shell: "mv {{ item }} {{ item|regex_replace('(.*)\\..*\\.(.*)', '\\1.\\2') }}"
      args:
        chdir: "{{ virtualenvDirectory }}/lib/{{ altInterpreterFileName }}/site-packages/"
      with_items: "{{ python3AptShareableObjectsFilenames }}"

Greetings

@rdonkin
Copy link

rdonkin commented Aug 17, 2020

@finswimmer - agree this is an upstream issue, though it's useful to have the workarounds here as they are not easy to do.

@refulgent-de - glad this was useful. I tend to do this when bootstrapping an Ansible laptop, but your Ansible code will be useful for other cases.

@duke66
Copy link

duke66 commented Apr 23, 2021

I also bump into the same issue but found an easy way to fix it.
For example, it is requirements.txt which can be run by pip install -r requirements.txt

sphinx
# apt binding for ansible from debian:
git+https://salsa.debian.org/python-debian-team/python-debian@master
git+https://salsa.debian.org/python-team/modules/python-distutils-extra@master
# use the branch for the debian/ubuntu you have
git+https://salsa.debian.org/apt-team/python-apt@1.1.y-xenial

@Brunux
Copy link

Brunux commented Aug 14, 2022

this is kind of old but I solve it by installing apt-get install python-apt-dev and pip package git+https://salsa.debian.org/apt-team/python-apt.git good luck

Copy link

github-actions bot commented Mar 1, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants