Upgrading GlobaLeaks to 2.60.130 prevent application from starting due to dependency version mismatch

[fpietrosanti]

Upgrading GlobaLeaks to 2.60.130 prevent application from starting due to dependency version mismatch.

It has been fixed by modifying the file requirements.txt in /usr/share/globaleaks/backend/

diff /usr/share/globaleaks/backend/requirements.txt /usr/share/globaleaks/backend/requirements.txt.orig
3c3

< cffi==1.3.1

cffi==0.8.2
6c6

< enum34==1.1.1

enum34==0.9.23

ERROR reported

• Starting GlobaLeaks daemon globaleaks * Enabling GlobaLeaks Apparmor Sandboxing... Some GlobaLeaks requirements are unmet

Unmet requirements:
cffi==0.8.2 [ Installed 1.3.1 ]
enum34==0.9.23 [ Installed 1.1.1 ]

The problem can be solved by:

1. Following the guidelines at https://github.com/globaleaks/GlobaLeaks/wiki
2. Installing missing requirements using rm -rf /tmp/pip-build-root/ && pip install -r /usr/share/globaleaks/backend/requirements.txt
   [fail]

[evilaliv3]

Got the point, but the fix is not correct;

the fix makes globaleaks working now, but will probable cause mess when the proper fix will be out.

[evilaliv3]

Ok got the point.
The issue is pip not being able to uninstall packages correctly as for: pypa/pip#2348

this has been refectored entirely and fixed in latest pip versions starting from 6.1.0: pypa/pip@c509c5c

i would suggest that for the distributions for which we have still to use pip (all before xenial and stretch) we always keep pip up to date to the latest.
the altenative is to run globaleaks inside a virtualenv, and at each upgrade delete the virtualenv and recreate it clean. this also would remove the issue of having to upgrade stuff.

here follows the console output that proves the bug:

Only starting from xenial and stretch having all packaged and stopping using pip we will be then safe.

pip install cffi==0.8.2
Downloading/unpacking cffi==0.8.2
  Downloading cffi-0.8.2.tar.gz (197kB): 197kB downloaded
  Running setup.py (path:/tmp/pip_build_root/cffi/setup.py) egg_info for package cffi

Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python2.7/dist-packages (from cffi==0.8.2)
Installing collected packages: cffi
  Found existing installation: cffi 1.3.1
    Can't uninstall 'cffi'. No files were found to uninstall.
  Running setup.py install for cffi
    building '_cffi_backend' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/c/_cffi_backend.o -lffi -o build/lib.linux-x86_64-2.7/_cffi_backend.so

Successfully installed cffi
Cleaning up...
root@(none):~# pip install cffi==1.3.1
Requirement already satisfied (use --upgrade to upgrade): cffi==1.3.1 in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python2.7/dist-packages (from cffi==1.3.1)
Cleaning up...

and this is the deployment machine that you know where you can see the issue affecting also pycparser

root@(none):/usr/local/lib/python2.7/dist-packages# ls
OpenSSL                     cryptography                       gnupg.pyc                   pkg_resources                      rstr-2.2.3.egg-info          storm                             zope.component-4.0.2-py2.7-nspkg.pth
Parsley-1.2-py2.7.egg-info  cryptography-0.8.2-py2.7.egg-info  idna                        pyOpenSSL-0.14-py2.7.egg-info      scrypt-0.6.1-py2.7.egg-info  storm-0.19-py2.7.egg-info         zope.component-4.0.2-py2.7.egg-info
Twisted-13.2.0.egg-info     cyclone                            idna-2.0.dist-info          pyasn1                             scrypt.py                    terml                             zope.event-4.0.1-py2.7-nspkg.pth
_cffi_backend.so            cyclone-1.1.egg-info               ipaddress-1.0.15.dist-info  pyasn1-0.1.7-py2.7.egg-info        scrypt.pyc                   transaction                       zope.event-4.0.1-py2.7.egg-info
_markerlib                  easy_install.py                    ipaddress.py                pycparser                          setuptools                   transaction-1.1.1-py2.7.egg-info  zope.interface-4.0.5-py2.7-nspkg.pth
_scrypt.so                  easy_install.pyc                   ipaddress.pyc               pycparser-2.10-py2.7.egg-info      setuptools-19.2.dist-info    twisted                           zope.interface-4.0.5-py2.7.egg-info
cffi                        enum                               ometa                       pycparser-2.14-py2.7.egg-info      six-1.5.2.dist-info          txsocksx
cffi-0.8.2-py2.7.egg-info   enum34-0.9.23-py2.7.egg-info       parsley.py                  python_gnupg-0.3.6-py2.7.egg-info  six.py                       txsocksx-1.13.0.0-py2.7.egg-info
cffi-1.3.1-py2.7.egg-info   gnupg.py                           parsley.pyc                 rstr                               six.pyc

[evilaliv3]

@xavfernandez can i please ask you a suggestion on the following issue? (i don't want to steal you so much time so here is simply a resume of what we are doing and i kindly ask you to give you a suggestion in implemeneint someting failure proof):

• we are building globaleaks that delivered as debian package, and in the end would like to be an official debian package (in stretch/xenial).
• currently not all globaleaks dependencies are packaged so that from the debian package we dirtily use pip and at every apt-get upgrade globaleaks we run an "pip install -r /usr/share/globaleaks/backend/requirements.txt"

but this seems to cause the same issue that you fixed in pypa/pip@c509c5c

do you have different suggestion for us to be sure that the enviroment is always update without failures of such a kind?

thanks you so much for your time

[xavfernandez]

I'm not sure what your issue is exactly...

Found existing installation: cffi 1.3.1
Can't uninstall 'cffi'. No files were found to uninstall.

This is strange, how did you install cffi ?

[evilaliv3]

@xavfernandez we have simply a file requirements.txt and the deb package does a pip install -r requirements.txt at every update.

what is happening is that pip fails to find the disinstallation files but anyhow install the new specified version so that if i do manually:
pip install package==versionX
pip install package==versionY
pip install package==versionZ

i will find installed the last version (versionZ) but there will be still present X, Y, Z eggs

[xavfernandez]

With what version pip do you observe such behavior ?

i will find installed the last version (versionZ) but there will be still present X, Y, Z eggs

What does python -c "from pkg_resources import Environment;print(Environment()['your_package'])"give you between each of these pip install ?

[evilaliv3]

pip --version
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)

python -c "from pkg_resources import Environment;print(Environment()['cffi'])"
[cffi 1.0.1 (/usr/local/lib/python2.7/dist-packages), cffi 0.8.2 (/usr/local/lib/python2.7/dist-packages)]

lsb_release -r -c -d
Description:    Ubuntu 14.04.3 LTS
Release:    14.04
Codename:   trusty

as you can see it says both are installed simply because if finds two eggs directories, but on the system there is only one "cffi" source directory and it corresponds to the latest version that one install so that if i do:
pip install package==X
pip install package==Y
pip install package==Z
i will hava the system thinking all X, Y, Z exist even if the only present is the Z
my doubt is then also that withouth the uninstall process the directory that i expect containing version Z probably is a mixture of files overwritten/left by X->Y->Z

let me know if i can help you somehow providing more feedback

[xavfernandez]

pip 1.5.4 is quite old, 7.1.2 is currently the latest, you could try upgrading.

Might be worth trying to uninstall the multiple installed package (in your case cffi) multiple times until everything is removed (i.e. running pip uninstall cffi and again pip uninstall cffi, etc).
Once everything is cleanup up, subsequent installation should then work.

You would then have to figure out what command makes things go bad.

[evilaliv3]

@xavfernandez: the main issue is that as you can see above, it is not possible to uninstall/reinstall the missing packages because strangely in this condition (the full log is above) exit saying Can't uninstall 'cffi'. No files were found to uninstall. even if two versions are installed instead.

[xavfernandez]

Well, how is the first cffi installed ? It seems it is lacking basic metadata to help pip uninstall it (like the list of installed files).

[evilaliv3]

all has been installed using pip via a pip install -r requirements.txt

the failure started to happening after having done a pip intall --upgrade -r requirements.txt

[evilaliv3]

The issue has not been reproduced since various releases so that this ticket can be probably closed.