UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-79: character maps to <undefined>

[engnadeau]

• safety version: 1.7.0
• Python version: Python 3.6.1
• Operating System: Windows-10-10.0.16299-SP0, AMD64

Description

• Trying to use safety check
• Same error always results: UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-79: character maps to <undefined>

What I Did

safety check -r simple-requirements.txt

Contents of simple-requirements.txt

safety

• There are absolutely no unicode characters in this file

Traceback

$ safety check -r simple-requirements.txt
Warning: unpinned requirement 'safety' found in simple-requirements.txt, unable to check.
Traceback (most recent call last):
  File "c:\users\nicholas\appdata\local\programs\python\python36\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\nicholas\appdata\local\programs\python\python36\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\nicholas\.virtualenvs\pybotics-d30fj9Hx\Scripts\safety.exe\__main__.py", line 9, in <module>
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\safety\cli.py", line 66, in check
    key=key
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\termui.py", line 420, in secho
    return echo(style(text, **styles), file=file, nl=nl, err=err, color=color)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\site-packages\click\utils.py", line 259, in echo
    file.write(message)
  File "c:\users\nicholas\.virtualenvs\pybotics-d30fj9hx\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-79: character maps to <undefined>

Similar Issues

• UnicodeEncodeError #22

[jayfk]

That's odd. I tried to reproduce this locally on different windows environments but wasn't able to.

It looks like the output is the problem here: https://github.com/pyupio/safety/blob/master/safety/formatter.py#L48

[jayvdb]

This is probably a dup of #22

and is likely fixed by #36

[jayvdb]

@nnadeau, is this fixed for you now?

[engnadeau]

I'll check asap when I get access to a Windows PC

[rahulrajaram]

My Jenkins Windows builds fail with this error only against Python 3.5-3.7. This is not an issue with Python 2.7 or Python 3.4.

I am unable to reproduce this issue on the same machine using the same Python executables as a regular user.

[leowonglaw]

I had the same issue when I used git bash insted of CMD. With CMD it works fine.

[Midnighter]

I'm seeing this error using Miniconda on AppVeyor. See the build log.

You can inspect the repo and configuration here.

[radeklat]

I'm hitting the same issue on Python 3.7.4 when running MinGW (GitBash) and also in cygwin bash but in that instance only when running it in a sub-shell as $(safety check -r requirements.txt)

[shahha]

In my case, export PYTHONIOENCODING=UTF-8 solved my issue.

[guidoavvisati]

Still not fixed, but the workaround from @shahha works for me as well

[rafaelpivato]

I know this has been around for quiet a while now, but I would need another feedback from anyone who is still experiencing this with latest version. If you do, please write me down your Safety version, your operating system and your complete stack trace.

[janusheide]

Still experience this error

Using safety 1.8.7
On a github windows-latest runner (https://github.com/actions/virtual-environments/blob/master/images/win/Windows2019-Readme.md)

I get the following:

Traceback (most recent call last):
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\hostedtoolcache\windows\Python\3.7.6\x64\Scripts\safety.exe\__main__.py", line 7, in <module>
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\safety\cli.py", line 89, in check
    click.secho(output_report, nl=False if bare and not vulns else True)
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\click\termui.py", line 548, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\click\utils.py", line 272, in echo
    file.write(message)
  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-79: character maps to <undefined>

[rafaelpivato]

@janusheide thank you very much for such details. That confirmed my hypothesis. This stack trace confirms the error was dumping a report to standard output:

  File "c:\hostedtoolcache\windows\python\3.7.6\x64\lib\site-packages\safety\cli.py", line 89, in check
    click.secho(output_report, nl=False if bare and not vulns else True)

Our JSON Formater is using a special character to draw the framing that is not being mapped properly. This console session shows the issue:

In [1]: import codecs                                                                                                      

In [2]: codecs.encode('═' * 7, encoding='cp1252')                                                                          
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-2-4656467a13d4> in <module>
----> 1 codecs.encode('═' * 7, encoding='cp1252')

/usr/lib/python3.7/encodings/cp1252.py in encode(self, input, errors)
     10 
     11     def encode(self,input,errors='strict'):
---> 12         return codecs.charmap_encode(input,errors,encoding_table)
     13 
     14     def decode(self,input,errors='strict'):

UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-6: character maps to <undefined>

So, we need some code that will either decode properly or just change that ASCII art to not use non-ASCII characters. My preference is towards the second option, which should cause less issues.

[janusheide]

Thanks for you quick response and shedding some light on the problem! To me just using a conforming character is the simpler, faster and more future prof approach, so that would get my vote :)

If there is something I can do to help, please let me know.