The 'ndg-httpsclient' distribution was not found

[kierun]

I run the following from cron every month, and got this error:

/opt/letsencrypt/letsencrypt-auto certonly --quiet --standalone --renew-by-default -d example.org -d www.example.org  
Creating virtual environment...
Installing Python packages...     
Installation succeeded.               
An unexpected error occurred:             
DistributionNotFound: The 'ndg-httpsclient' distribution was not found and is required by requests                               
Please see the logfile 'certbot.log' for more details.

Clearly, "example" is not my domain, I used the correct one. The certbot.log does not offer any more details:

Traceback (most recent call last):
  File "/root/.local/share/letsencrypt/bin/letsencrypt", line 11, in <module>
    sys.exit(main())
  File "/opt/letsencrypt/certbot/main.py", line 858, in main
    plugins = plugins_disco.PluginsRegistry.find_all()
  File "/opt/letsencrypt/certbot/plugins/disco.py", line 183, in find_all
    plugin_ep = PluginEntryPoint(entry_point)
  File "/opt/letsencrypt/certbot/plugins/disco.py", line 34, in __init__
    self.plugin_cls = entry_point.load()
  File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2201, in load
    self.require(*args, **kwargs)
  File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2218, in require
    items = working_set.resolve(reqs, env, installer)
  File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 830, in resolve
    raise DistributionNotFound(req, requirers)
DistributionNotFound: The 'ndg-httpsclient' distribution was not found and is required by requests

What am I missing?

System is Ubuntu 14.04.5 LTS (Trusty Tahr) if that matters and certbot is the lastest master as of today.

[bmw]

That's an interesting error. Can you tell me the version of your letsencrypt-auto script? Probably the easiest way for you to do this is grep -m1 LE_AUTO_VERSION /opt/letsencrypt/letsencrypt-auto. If there's no output from grep, you have a very old letsencrypt-auto and I encourage you to download a new one from https://dl.eff.org/certbot-auto.

To try and resolve the problem, I recommend moving/deleting the /root/.local/share/letsencrypt directory and try running letsencrypt-auto again. I haven't seen any instances of this before, but it's possible the files there got corrupted and simply moving this directory and running the script again will solve the problem.

[kierun]

That's an interesting error.

Well, that is good! ☺ I like being helpful.

(master|✓); grep -m1 LE_AUTO_VERSION /opt/letsencrypt/letsencrypt-auto
26:LE_AUTO_VERSION="0.10.2"
(master|✓); git log |head -3
commit 20be8b327d5cb19b2ef5dedae3b771c1de952bd3
Author: Brad Warren <bmw@users.noreply.github.com>
Date:   Tue Jan 31 17:08:21 2017 -0800

Removing the /root/.local did not have any effects but that pointed me in the right direction. I used sudo zsh to get a root shell. Which, of course, kept my PYTHONPATH set to /home/usr/.local so that's where things started to go wrong. Un-setting the PYTHONPATH worked just fine. Oh, I had to stop nginx from running as well.

So, feel free to close this with a error between keyboard and floor. Although it might be worth adding a note in the documentation just in case someone else stumbles into this.

Thanks!

[bmw]

Ideally I'd like to fix the problem in letsencrypt-auto to completely prevent issues for users in the future, however, we should add documentation if that's not possible for some reason.

I'm not sure I completely understand what the problem was though. Can you elaborate a bit? What was the problem with your PYTHONPATH being set to /home/usr/.local?

Oh, I had to stop nginx from running as well.

Depending on your setup, you may be able to avoid this in the future by using our nginx or webroot plugins.

[kierun]

I'm not sure I completely understand what the problem was though.

I tend to use sudo a lot. Generally, when I want to issue more than one command, I end up doing sudo zsh which runs a z-shell as root but keeps my settings -- such as ls options, etc… In most cases, this is fine. However, in this case the PYTHONPATH was set to include my own ~/.local which contains various stuff I use as a user but that interfered with root.

Does that make sense?

Depending on your setup, you may be able to avoid this in the future by using our nginx or webroot plugins.

Super! Thank you very much.

[bmw]

Does that make sense?

It does! Thanks.

This doesn't seem like something letsencrypt-auto can really programmatically work around as we can't really ignore a user's PYTHONPATH if it is set. What kind of note in the documentation would you like to add? Something under info about certbot-auto like:

"Note: If you've configured any custom Python environment variables on your system, make sure they are properly set when running certbot-auto and sudo."

[kierun]

"Note: If you've configured any custom Python environment variables on your system, make sure they are properly set when running certbot-auto and sudo."

This would be great. I do not think it need more than that.

Do you want me to do a PR?

Indeed, I 100% agree with all of what you said. It was my broken system but it's a system others use too so others might fall into that trap. Kinda belts and braces.

[bmw]

Do you want me to do a PR?

If you're interested, that'd be great! The relevant file to modify is docs/install.rst in the root of our repo. It probably makes the most sense to put it under certbot-auto or Problems with Python virtual environment. After following our developer setup instructions, you can find instruction about how to build the docs after modifying them here.

I'd also expand my proposed text above to say "...certbot-auto and su/sudo". Certbot falls back to su if sudo is unavailable on your system.

[kierun]

PR added. Hope it is okay.

[jsha]

I'm not convinced that this is an issue specific to sudo zsh (or sudo bash). I think most likely it's related to cron jobs not having the same environment variables set as interactive shells, a longstanding problem with Certbot and cron jobs.

@kierun, could you post your whole crontab, and tell us what user it runs as?

[kierun]

@jsha Here it is. Mostly, it's the default Fedora one with certbot added.

# /etc/crontab: system-wide crontab                               
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
                                                  
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
          
# m h dom mon dow user  command
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )    
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )      
13 4    * * *   root    /opt/letsencrypt/certbot-auto renew --no-self-upgrade --nginx                        
@monthly root /opt/letsencrypt/letsencrypt-auto certonly --quiet --standalone --renew-by-default -d example.org -d www.example.org >> /var/log/letsencrypt/letsencrypt-auto-update.log

[jsha]

Thanks! This is interesting:

13 4 * * * root /opt/letsencrypt/certbot-auto renew --no-self-upgrade --nginx
@monthly root /opt/letsencrypt/letsencrypt-auto certonly --quiet --standalone --renew-by-default -d example.org -d www.example.org >> /var/log/letsencrypt/letsencrypt-auto-update.log

It looks like you've got two cron jobs, one running certbot-auto with --no-self-upgrade and one running letsencrypt-auto, with self-upgrade defaulted on. Is that intentional?

Also, you've got --renew-by-default on, which is inadvisable: It will renew your certs on every run, rather than just if they are close to expiry.

Are you still getting the original error in this thread? Are you getting it by email or from the log? Was the error definitely from the monthly letsencrypt-auto run, or could it have been from the certbot-auto run?

Do you see the same errors reliably in /var/log/letsencrypt.log for the past few days?

[kierun]

It looks like you've got two cron jobs […]

You are right! Silly me. I was meant to delete one of those.

Are you still getting the original error in this thread?

No, I am not getting the error using the "fix" I detailed in the PR. ☺

[jsha]

I'm inclined to close the PR unless we can come up with a specific set of steps to reproduce the issue. I've tried reproducing with, e.g. PYTHONPATH=/foo/bar ./certbot-auto --help and can't. Thoughts? @bmw?

[kierun]

@jsha That is really weird. But if you cannot reproduce it, then it must have been something more subtle in my set up that got fixed as a side effect of resetting the PYTHONPATH. Hum…

[jsha]

Thanks! Closing for now, reopen or open a new ticket if you find more problems. Thanks!