Update fails on Ubuntu 20.04 - virtualenv cleanup missing

[MrFly]

The update of the client / server fails on Ubuntu. Since 0.4.x I always have to remove /var/lib/little-brother/vitualenv/. I did this meanwhile six times from 0.4.9 to 0.4.11 (+ some times I did not think of it and first tried without deleting).

It would be great, if the installer package could do this job as some pre-cleanup. It's sure not a big problem, already documented as workaround/problem solution and just one thing to do for the user. But it would reduce the admin's time for updates and fail rates.

[marcus67]

Hmmm. This is strange. This error should only occur when the version of Python has changed since the most recent install resulting in a mismatch of Python, pip, and the setup-tools. As a rule the virtual environment can be re-used for the updated version and does not have to be removed. I have not encountered this problem for quite a few installs now. It may be dependent on the distribution. Maybe is does not occur on Debian that easily though I have no clue why this should be the case.

Question: After you removed the virtual environment did apt-get install python3-setuptools python3-pip actually install new versions or did apt-get say that all packages are up-to-date?

[MrFly]

I do an update/upgrade every few days, but only apt, not the mentioned packages. But these packages were already installed before I installed the deb-package via dpkg. I don't think, that these were updated every time I installed the deb, because I did not always update Ubuntu before installing.

But I remember that at least setuptools was built/downloaded/something else in the installation process if I installed the deb after removing the virtenv.

[marcus67]

I just tried an update using a clean Ubuntu 20.04 VM after installation of a package list sent in by user Rosy. She also reported a problem with updating from 0.4.12 to 0.4.14. The error messages did not conclusively hint at the virtual environment but after removing it the update went through smoothly. These were the error messages:

Building wheels for collected packages: little-brother, python-base-app
  Building wheel for little-brother (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /var/lib/little-brother/virtualenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-khphdohp/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-khphdohp/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-tlmai9r0
       cwd: /tmp/pip-req-build-khphdohp/

and

copying little_brother/alembic.ini -> build/lib/little_brother
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-khphdohp/setup.py", line 110, in <module>
      setup(**setup_params)
    File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/command/build_py.py", line 53, in run
      self.build_package_data()
    File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/command/build_py.py", line 126, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for little-brother

and

Building wheel for python-base-app (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /var/lib/little-brother/virtualenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ebnuti0v/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ebnuti0v/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-sn175itq
       cwd: /tmp/pip-req-build-ebnuti0v/

and

 copying python_base_app/static/icons/icon-python-base-app-128x128.png -> build/lib/python_base_app/static/icons
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-ebnuti0v/setup.py", line 68, in <module>
      setup(**setup_params)
    File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/command/build_py.py", line 53, in run
      self.build_package_data()
    File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/command/build_py.py", line 126, in build_package_data
      srcfile in self.distribution.convert_2to3_doctests):
  AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
  ----------------------------------------
  ERROR: Failed building wheel for python-base-app

and

   Running setup.py install for little-brother ... error
    ERROR: Command errored out with exit status 1:
     command: /var/lib/little-brother/virtualenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-khphdohp/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-khphdohp/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9zcbww73/install-record.txt --single-version-externally-managed --compile --install-headers /var/lib/little-brother/virtualenv/include/site/python3.8/little-brother
         cwd: /tmp/pip-req-build-khphdohp/

and

   copying little_brother/alembic.ini -> build/lib/little_brother
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-khphdohp/setup.py", line 110, in <module>
        setup(**setup_params)
      File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.8/distutils/command/install.py", line 589, in run
        self.run_command('build')
      File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/command/build_py.py", line 53, in run
        self.build_package_data()
      File "/var/lib/little-brother/virtualenv/lib/python3.8/site-packages/setuptools/command/build_py.py", line 126, in build_package_data
        srcfile in self.distribution.convert_2to3_doctests):
    AttributeError: 'Distribution' object has no attribute 'convert_2to3_doctests'
    ----------------------------------------
  Rolling back uninstall of little-brother

and finally

ERROR: Command errored out with exit status 1: /var/lib/little-brother/virtualenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-khphdohp/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-khphdohp/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9zcbww73/install-record.txt --single-version-externally-managed --compile --install-headers /var/lib/little-brother/virtualenv/include/site/python3.8/little-brother Check the logs for full command output.

.

[marcus67]

Some more information on this error. As far as I could find out, it goes like this: Ubuntu (and possibly other distributions) have a rather old version of the PIP package setuptools bundled with their release (e.g. 44.0.0). When LittleBrother is installed into the virtual environment its Debian post installation script updates the package setuptools (currently to 60.5 something). Upon the next upgrade of LittleBrother the old setuptools outside and the new package tools inside collide in some places. The details are beyond my understanding but looking at the stack traces above it is obvious that at one point the calling sequence leaves the scripts inside the environment and calls scripts outside the environment resulting in a clash.
This explains why deleting the virtual environment takes care of the problem. What also helps is to downgrade setuptools inside the environment or even uninstalling it completely. I have decided to remove the call to install/upgrade it in the post installation script. This seems to take care of it.

[marcus67]

The version 0.4.15 containing the fix for this issue has just been uploaded. Thanks for reporting this!