With 0.32.0, certbot-auto stopped working for some EOL distributions

[caplod]

OS: Debian Wheezy

Since yesterday certbot-auto stopped working with a message 'pip' is a package and cannot be directly executed

I ran this command:
certbot-auto --debug

output:

Bootstrapping dependencies for Debian-based OSes... (you can skip this with --no-bootstrap) Hit http://rspamd.com wheezy Release.gpg Hit http://rspamd.com wheezy Release Hit http://rspamd.com wheezy/main amd64 Packages Hit http://deb.freexian.com wheezy-lts Release.gpg Hit http://deb.freexian.com wheezy-lts-kernel Release.gpg Hit http://deb.freexian.com wheezy-lts Release Hit http://deb.freexian.com wheezy-lts-kernel Release Hit http://deb.freexian.com wheezy-lts/main amd64 Packages Hit http://deb.freexian.com wheezy-lts/contrib amd64 Packages Hit http://deb.freexian.com wheezy-lts/non-free amd64 Packages Hit http://deb.freexian.com wheezy-lts-kernel/main amd64 Packages Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done augeas-lenses is already the newest version. ca-certificates is already the newest version. libaugeas0 is already the newest version. libffi-dev is already the newest version. libssl-dev is already the newest version. openssl is already the newest version. python is already the newest version. python-dev is already the newest version. gcc is already the newest version. python-virtualenv is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. Creating virtual environment... Installing Python packages... /opt/eff.org/certbot/venv/bin/python: No module named pip.__main__; 'pip' is a package and cannot be directly executed Traceback (most recent call last): File "/tmp/tmp.mn0uym6MSb/pipstrap.py", line 177, in <module> sys.exit(main()) File "/tmp/tmp.mn0uym6MSb/pipstrap.py", line 149, in main pip_version = StrictVersion(check_output([python, '-m', 'pip', '--version']) File "/usr/lib/python2.7/subprocess.py", line 544, in check_output raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['/opt/eff.org/certbot/venv/bin/python', '-m', 'pip', '--version']' returned non-zero exit status 1

[RaphaelRochet]

Got the same, after an auto upgrade from 0.30.0 to 0.32.0

[tariver]

Second that, after upgrade from 0.31.0 to 0.32.0
Installing Python packages... /opt/eff.org/certbot/venv/bin/python: No module named pip.__main__; 'pip' is a package and cannot be directly executed Traceback (most recent call last): File "/tmp/tmp.AEi9JsFBtv/pipstrap.py", line 177, in <module> sys.exit(main()) File "/tmp/tmp.AEi9JsFBtv/pipstrap.py", line 149, in main pip_version = StrictVersion(check_output([python, '-m', 'pip', '--version']) File "/usr/lib/python2.7/subprocess.py", line 544, in check_output raise CalledProcessError(retcode, cmd, output=output) subprocess.CalledProcessError: Command '['/opt/eff.org/certbot/venv/bin/python', '-m', 'pip', '--version']' returned non-zero exit status 1
Debian 7.9 wheezy

[olemd]

Third that. Upgrade from 0.31.0 to 0.32.0.

Debian 7.11 wheezy

[RaphaelRochet]

Workaround for me :
Removing /opt/eff.org and reinstalling certbot-auto 0.31.0
(modified to prevent accidental upgrade, joined here)
certbot-auto.zip

[alexzorin]

You can also use --no-self-upgrade on the CLI against the 0.31 release:

wget https://raw.githubusercontent.com/certbot/certbot/75499277be6699fd5a9b884837546391950a3ec9/certbot-auto
chmod +x ./certbot-auto
./certbot-auto --no-self-upgrade

[olemd]

Seems to be changes in pipstrap that triggers this - manually changing pip-invocation from ['python', '-m', 'pip', … ] back to just ['pip', … ] has things working again.

Curiously it also adds

/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
  utils.DeprecatedIn23,

[problem9]

0.30, 0.31 is working for me. debian wheezy

[adferrand]

All issues seems to be related to wheezy (if different, please notice it). Indeed wheezy uses an ancient version of pip that cannot be run as a module (python -m pip).

However wheezy is EOL since end of 2018, and official support of Certbot has been dropped also.

One immediate solution is to upgrade to a non-EOL Debian version.

[olemd]

I don't have a system-installed pip - only the one that comes via the virtual environment in /opt/eff.org/. Though it also appears quite ancient:

$ /opt/eff.org/certbot/venv/bin/pip --version
pip 9.0.1 from /opt/eff.org/certbot/venv/local/lib/python2.7/site-packages (python 2.7)

I would, perhaps naively, have expected the virtenv to install a newer pip?

But yes, wheezy is EOLed.

[adferrand]

The pip that is failing is not coming from the existing venv of Certbot. It comes from a new venv generated by certbot-auto to upgrade everything.

If you try to create a bare venv from the installed virtualenv (with --no-download to avoid self-upgrade, see below), outside of certbot-auto, and ask its version, you will get an even more ancient one.

So in theory, if you could get a more recent version of the deb package python-virtualenv, it could solve the issue. But it is likely to break other things...

Yes you are right, normally virtualenv self-upgrade its base components. But we disabled it when one of the latest release of pip was broken, and so broke also certbot-auto... Now the pip installed in the new venv is pinned by pipstrap.

[adferrand]

After some tests, yes on wheezy, pip 1.1 is installed by default in venv. Call to python -m pip is supported starting to pip 1.2+.

Also, version of virtualenv in wheezy does not support the self-upgrading.

[spol50]

My case and solution:

• Debian 7.11 wheezy
• python2.7
• python-pip NOT installed

My steps:
#ln -fs /usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.py /usr/lib/python2.7/
#wget https://raw.githubusercontent.com/certbot/certbot/75499277be6699fd5a9b884837546391950a3ec9/certbot-auto
#chmod +x ./certbot-auto
#certbot-auto renew --no-self-upgrade

it download some files and works fine.

[ntabu]

Hi

I downloaded the latest version of the certbot (https://dl.eff.org/certbot-auto) and I changed the two lines that were problematic.

Line 1347 and 1357 :

pip_version = StrictVersion(check_output([python, '-m', 'pip', '--version'])
by
pip_version = StrictVersion(check_output(['pip', '--version'])

and command = [python, '-m', 'pip', 'install', '--no-index', '--no-deps', '-U']
by
command = ['pip', 'install', '--no-index', '--no-deps', '-U']

As specified by olemd,

Seems to be changes in pipstrap that triggers this - manually changing pip-invocation from ['python', '-m', 'pip', … ] back to just ['pip', … ] has things working again.

It works for a distrib 7.3 (debian)

Regards,
``

[richardloxley]

Having the same problem since this morning - Debian 7.11.

[frommelmak]

Same problem here trying to renew a cert in Ubuntu 12.04.4 LTS

[adferrand]

Ubuntu Precise is also EOL since 2017, and ships with pip 1.0.

[frommelmak]

thnxs @adferrand I know Ubuntu Precise has reach the End of Life but certbot-auto has been working properly on this server til now. Time to upgrade.

[yodog]

breaking things around here as well.
can not upgrade production server OS, so please rollback to the previous (working) state.

[adferrand]

As a small team of maintainers, we cannot support every distribution out there forever. Our limit is when a given version reaches its EOL. Indeed at this point it means that every dependency provided by the system will stop to be updated. It means also for Certbot that if one of this dependency start to break the software, there is nothing we can do about it from upstream.

Here the situation is that pip < 1.2, released almost 7 years ago, does not support what has become the recommended way to launch pip for self-update, as a module, using python -m pip. We followed this path, considering that every distribution we support and that is actively maintained ships with a sufficiently recent version of pip to use that.

It is true that certbot-auto failing suddendly because of that is a little harsh without a deprecation warning that could be displayed some time before the final break. We plan to improve that in the future.

The only sane way, regarding our capability of support, and the interest on user side to rely on software that are sufficiently maintained for features and security, is to upgrade the OS.

For people that are not in a situation of doing that, several solutions have been provided here to stick on a old version of certbot-auto that still works, and shutting down self upgrade. But in this case also, the only way in the long term is to upgrade in a way or another.

So I am closing the issue now, as we will not fix certbot-auto for EOL distributions.

[yodog]

@adferrand
I know this software demands a lot of time from the team, but when you create something big that thousands of people start using and depending on, you cant just break it and say 'thats it folks, update your production servers'.

we, as your users, work in companies of all sizes, and maybe cant afford to stop the environment and risk everything that is already working fine because of 2 lines of code.

imagine that instead of where you are right now, you had a company paying for this code.
and your boss asked what changed and how to fix it.
would you reply 'let me just change a couple lines' or 'its time to update all our servers'?

i know i have no right of complaining here since i dont contribute to the project, but still.
this is my rant.

[YetiRick]

@yodog
Also not a contributor, but I'll provide an answer since I've been exactly there. I would say, "It's time to update our servers. I can fix it temporarily, but it will break again, and with increasing frequency the further behind "current" we get."

If I were the devs for this project I would probably push out the two line rollback as .33 with a notice that proper calling procedures will be implemented in the first version released after 60 days have elapsed. I would not leave legacy code in place just to make people running dinosaur systems happy. That is worse for a greater number of users in the long run. It's EOL. You simply do not compromise your work to support EOL software and hardware. Not for a free, open-source product, anyway.

I agree that this could have been handled more cleanly, but then since this script tends to run automated by cron, even a notice output during the auto-upgrade process would be missed by 90% of the sysadmins I know. Most rely on expiration warning emails rather than examining the logfiles for runtime messages.

[wmakley]

Just hit this. Not understanding the source of the issue. Is it Python, Pip, or Virtualenv? Can't I just install a newer python or pip? Is there some reason I am stuck with with the system pip? I can't upgrade the server OS for a little while, but I do know how to install Python outside the purview of the package manager.

[BenjamenMeyer]

@adferrand Just FYI - it's not always easy to upgrade servers. In my particular case I have 2 cloud servers on Rackspace Public Cloud. To upgrade means rebuilding the server (spin up a new server, migrate everything over, change IPs, etc.) b/c in-place distro upgrades are not supported by the host. Needless to say, @ntabu's little fix did the trick for the server that didn't do the install.

FYI - you can replicate by simply certbox-auto --install-only which I was doing as I was upgrading from an old letsencrypt version to certbot due to the ACME change, only to find this issue. It would be helpful to at least point to @ntabu 's little change and say "folks, just go do this if you must, but we advise upgrading your OS ASAP".

@wmakley it's an interface change supported by newer versions of pip. The old method still works even on the newer versions. Not sure why folks are wanting to do python -m pip (new method) instead of just pip (old method still supported and not going anywhere).

[BenjamenMeyer]

Seems #6775 was the PR that broke the older distros

[adferrand]

I really understand your situation. I am myself also a Certbot user in a corporate environment. We largely use an ancient CentOS 6 version in a lot of our servers, that is (hopefully) still maintained until 2020. But large efforts are engaged to move on CentOS 7.

Because in my opinion, it is not acceptable to let production servers on an unsupported distribution, for which the EOL is known several years before. If in a corporate environment the cost to stay on a old, but supported, version of a software, is often lower than upgrading. However, things change with EOL: having a bug, or a critical security flaw not fixed, become the highest risk over everything else.

Let put things in another point of view. Let's suppose that Certbot would continue to work apparently for unsupported distributions. So far, so good, but nobody on our team continue to make regression/security tests on it. Then, something not as visible as a big break happens, like a hidden flaw in one of our third-party dependency. Then people complain on us to expose security flaw on their system. And then we could answer that it is not supported anymore, so we can not take responsibility for that. Formally yes it is true, but we would know that we let intentionally Certbot running on systems where the security could be broken.

It is quite the opposite of our purpose, isn't ? Nobody here would take this position knowing that.

At least, here the situation is clear. Most of unsupported EOL distributions will not run Certbot out-of-the-box, you will not get, without knowing it, some kind of Certbot zombie that would continue to work in an unknown sanity state.

That said, you still can fix yourself certbot-auto to make the shift until your production system is updated. See @ntabu answer. Add to it the --no-self-upgrade to avoid self-upgrade. See also solution from @alexzorin, or @RaphaelRochet. It will gives you immediate mitigation.

Note that I cannot decently recommend to trigger pip self-upgrade like @wmakley proposed. In such old distributions, replacing os systems packages by the most recent ones from PyPI, cannot do anything else than breaking your Python environment.

As a final note, I think that any further support should definitively profit to every community member that would be in this situation. So I will stop to continue to answer here, and invite you to continue the discussion, if required, on Let'sEncrypt support forum.

[arcataroger]

So sad, just installed certbot-auto a few days ago and then this happened :( --no-self-upgrade doesn't work either.

Edit: Works now... had to delete 0.32 and redownload 0.31 for --no-self-upgrade to work. Thanks for the workaround.

[adferrand]

Related topics on Letsencrypt forum:

• https://community.letsencrypt.org/t/pip-error-with-certbot-auto/88200