Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeDecodeError: 'utf-8' codec can't decode #5646

Closed
mashirozx opened this issue Mar 1, 2018 · 12 comments
Closed

UnicodeDecodeError: 'utf-8' codec can't decode #5646

mashirozx opened this issue Mar 1, 2018 · 12 comments

Comments

@mashirozx
Copy link

@mashirozx mashirozx commented Mar 1, 2018

If you're having trouble using Certbot and aren't sure you've found a bug or
request for a new feature, please first try asking for help at
https://community.letsencrypt.org/. There is a much larger community there of
people familiar with the project who will be able to more quickly answer your
questions.

My operating system is (include version):

Ubuntu 17.10
Nginx 1.12.1
Certbot 0.21.1

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install python-certbot-nginx 

I ran this command and it produced this output:

# sudo certbot --nginx certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 650: invalid continuation byte
Please see the logfiles in /var/log/letsencrypt for more details.

Certbot's behavior differed from what I expected because:

Seems the same issue in Apache in #5211 , not sure it's fixed? What's more, my server is Nginx, I do use some utf-8 words (Chinese, and all they are in comment after # tag) in my Nginx conf files.

I was using the older version of Certbot before, and it worked fine, today I find it cannot obtain new certificates and so installed the new version and got this problem.

Here is a Certbot log showing the issue (if available):

Logs are stored in /var/log/letsencrypt by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

The log:

2018-03-01 18:57:19,455:DEBUG:certbot.main:certbot version: 0.21.1
2018-03-01 18:57:19,457:DEBUG:certbot.main:Arguments: ['--nginx']
2018-03-01 18:57:19,457:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-03-01 18:57:19,469:DEBUG:certbot.log:Root logging level set at 20
2018-03-01 18:57:19,470:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-03-01 18:57:19,473:DEBUG:certbot.plugins.selection:Requested authenticator nginx and installer nginx
2018-03-01 18:57:19,855:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.21.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1240, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 981, in run
    installer, authenticator = plug_sel.choose_configurator_plugins(config, plugins, "run")
  File "/usr/lib/python3/dist-packages/certbot/plugins/selection.py", line 189, in choose_configurator_plugins
    authenticator = installer = pick_configurator(config, req_inst, plugins)
  File "/usr/lib/python3/dist-packages/certbot/plugins/selection.py", line 25, in pick_configurator
    (interfaces.IAuthenticator, interfaces.IInstaller))
  File "/usr/lib/python3/dist-packages/certbot/plugins/selection.py", line 77, in pick_plugin
    verified.prepare()
  File "/usr/lib/python3/dist-packages/certbot/plugins/disco.py", line 248, in prepare
    return [plugin_ep.prepare() for plugin_ep in six.itervalues(self._plugins)]
  File "/usr/lib/python3/dist-packages/certbot/plugins/disco.py", line 248, in <listcomp>
    return [plugin_ep.prepare() for plugin_ep in six.itervalues(self._plugins)]
  File "/usr/lib/python3/dist-packages/certbot/plugins/disco.py", line 130, in prepare
    self._initialized.prepare()
  File "/usr/lib/python3/dist-packages/certbot_nginx/configurator.py", line 131, in prepare
    self.parser = parser.NginxParser(self.conf('server-root'))
  File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 38, in __init__
    self.load()
  File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 45, in load
    self._parse_recursively(self.config_root)
  File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 66, in _parse_recursively
    self._parse_recursively(subentry[1])
  File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 61, in _parse_recursively
    self._parse_recursively(entry[1])
  File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 56, in _parse_recursively
    trees = self._parse_files(filepath)
  File "/usr/lib/python3/dist-packages/certbot_nginx/parser.py", line 206, in _parse_files
    parsed = nginxparser.load(_file)
  File "/usr/lib/python3/dist-packages/certbot_nginx/nginxparser.py", line 123, in load
    return loads(_file.read())
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 650: invalid continuation byte
2018-03-01 18:57:19,861:ERROR:certbot.log:An unexpected error occurred:

Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:

My nginx conf worked well before and I didn't modify it, sure not conf problem.

Thanks! :)

@bmw

This comment has been minimized.

Copy link
Member

@bmw bmw commented Mar 2, 2018

Hey @mashirozx. Thanks a lot for the detailed bug report.

I think this is closely related to #5236 and #5337. While there are some slight differences, the traceback is almost identical to the one in #5236. This might be fixed by #5341.

We'll try and look into this soon, but we'd gladly take help around this issue.

@bmw

This comment has been minimized.

Copy link
Member

@bmw bmw commented Mar 5, 2018

I wonder if this is due to Debian and the Ubuntu PPA switching Certbot to Python 3.

@Zopieux

This comment has been minimized.

Copy link

@Zopieux Zopieux commented May 2, 2018

As noted elsewhere, this is because nginx mime.types contains which cannot be parsed as ASCII.

I could not reproduce the bug in a development setup, only with some remote server with a "rawer" configuration. In fact, because I was missing some configuration, I was using the C locale and Python uses ASCII decoder when locale is C.

The solution is to generate an UTF-8 locale, such as en_US.UTF-8, and launch certbot with it:

LANG=en_US.UTF-8 certbot […]

Pinging @bmw as this looks like an issue caused by users' setup, not certbot itself.

@SteveParson

This comment has been minimized.

Copy link

@SteveParson SteveParson commented Aug 17, 2018

Just confirming the above. As @Zopieux said, setting the locale correctly will solve the problem.

@egberts

This comment has been minimized.

Copy link

@egberts egberts commented Aug 26, 2018

Using the following command:

grep -r -P '[^\x00-\x7f]' /etc/apache2 /etc/letsencrypt /etc/nginx

Found mine in

/etc/letsencrypt/options-ssl-nginx.conf:        # The following CSP directives don't use default-src as 

Using shed, I found the offending sequence. It turned out to be an editor mistake.

00008099:     C2  194 302 11000010
00008100:     A0  160 240 10100000
00008101:  d  64  100 144 01100100
00008102:  e  65  101 145 01100101
00008103:  f  66  102 146 01100110
00008104:  a  61  097 141 01100001
00008105:  u  75  117 165 01110101
00008106:  l  6C  108 154 01101100
00008107:  t  74  116 164 01110100
00008108:  -  2D  045 055 00101101
00008109:  s  73  115 163 01110011
00008110:  r  72  114 162 01110010
00008111:  c  63  099 143 01100011
00008112:     C2  194 302 11000010
00008113:     A0  160 240 10100000
@aral

This comment has been minimized.

Copy link

@aral aral commented Aug 27, 2018

Ubuntu 18.04, locale set to en_US.UTF-8, and just in case setting the env var:

LANG=en_US.UTF-8 sudo ./certbot-auto --nginx

Error:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for archive.better.fyi
Cleaning up challenges
An unexpected error occurred:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 17: ordinal not in range(128)
Please see the logfiles in /var/log/letsencrypt for more details.

Details:

ubuntu@better-archive:~/letsencrypt$ sudo tail /var/log/letsencrypt/letsencrypt.log
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 976, in save
    self.parser.filedump(ext='')
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/parser.py", line 243, in filedump
    out = nginxparser.dumps(tree)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/nginxparser.py", line 134, in dumps
    return str(RawNginxDumper(blocks.spaced))
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/nginxparser.py", line 98, in __str__
    return ''.join(self)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 17: ordinal not in range(128)
@Zopieux

This comment has been minimized.

Copy link

@Zopieux Zopieux commented Sep 1, 2018

@aral did you actually enabled/generated the en_US.UTF-8 locale? Check /etc/locale.gen and $ localectl list-locales.

@neilpd

This comment has been minimized.

Copy link

@neilpd neilpd commented Sep 10, 2018

I suspect I am having a similar issue.

Ubuntu 16.04
Apache 2.4.18
Certbot 0.26.1-1+ubuntu16.04

root@rsl-wordpress:/#  LANG=en_US.UTF-8 certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): [redacted]
An unexpected error occurred:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 4: invalid continuation byte
Please see the logfiles in /var/log/letsencrypt for more details.
root@rsl-wordpress:/# sudo tail /var/log/letsencrypt/letsencrypt.log 
  File "/usr/lib/python3/dist-packages/certbot/display/ops.py", line 50, in get_email
    force_interactive=True)
  File "/usr/lib/python3/dist-packages/certbot/display/util.py", line 185, in input
    ans = input_with_timeout(message)
  File "/usr/lib/python3/dist-packages/certbot/display/util.py", line 88, in input_with_timeout
    line = rlist[0].readline()
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 4: invalid continuation byte
2018-09-10 09:45:04,011:ERROR:certbot.log:An unexpected error occurred:
@egberts

This comment has been minimized.

Copy link

@egberts egberts commented Sep 10, 2018

Ultra high probability is that someone's editor introduced the artifact. Simplest thing to do is to re-edit the file and remove the offending character before saving and closing the text file.

@Chickensoupwithrice

This comment has been minimized.

Copy link

@Chickensoupwithrice Chickensoupwithrice commented Sep 14, 2018

I had this error but it was due to odd characters in the comments of the nginx config file for the site I was trying to get a certificate for.

Following @egberts advice and command:

grep -r -P '[^\x00-\x7f]' /etc/apache2 /etc/letsencrypt /etc/nginx

And then deleting the offending lines (since they were comments it didn't matter) solved the issue.
Thanks for your help @egberts! 👍

@Zopieux

This comment has been minimized.

Copy link

@Zopieux Zopieux commented Sep 15, 2018

Batch-editing random conf files without understanding the underlying issue, what could possibly go wrong? Let's fix the symptoms instead of getting to the root issue which has already been identified in this thread!

Again, certbot modifications shall be transparent. However broken they seem, if the conf files work for their target software (nginx, apache, whatever), certbot should not die trying to modify them. Or at least it should die with a better explanation.

@ohemorange

This comment has been minimized.

Copy link
Contributor

@ohemorange ohemorange commented Sep 18, 2018

Closing in favor of duplicate #5337, which has a PR at #5341.

@ohemorange ohemorange closed this Sep 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.