Flycheck checker python-flake8 returned non-zero exit code 1, but its output contained no errors

[galeo]

When checking a python3 file whose code contains unicode characters, the python-flake8 checker gets errors. Here is the flycheck-compile output:

-*- mode: compilation; default-directory: "~/Desktop/" -*-
Compilation started at Thu Mar  9 23:37:19

flake8 --format\=default --stdin-display-name /Users/galeo/Desktop/flycheck-error.py - < /Users/galeo/Desktop/flycheck-error.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flake8/checker.py", line 471, in run_ast_checks
    ast = self.processor.build_ast()
  File "/usr/local/lib/python3.6/site-packages/flake8/processor.py", line 211, in build_ast
    return compile(''.join(self.lines), '', 'exec', PyCF_ONLY_AST)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 9-10: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/flake8", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/flake8/main/cli.py", line 16, in main
    app.run(argv)
  File "/usr/local/lib/python3.6/site-packages/flake8/main/application.py", line 328, in run
    self._run(argv)
  File "/usr/local/lib/python3.6/site-packages/flake8/main/application.py", line 316, in _run
    self.run_checks()
  File "/usr/local/lib/python3.6/site-packages/flake8/main/application.py", line 246, in run_checks
    self.file_checker_manager.run()
  File "/usr/local/lib/python3.6/site-packages/flake8/checker.py", line 319, in run
    self.run_serial()
  File "/usr/local/lib/python3.6/site-packages/flake8/checker.py", line 303, in run_serial
    checker.run_checks()
  File "/usr/local/lib/python3.6/site-packages/flake8/checker.py", line 573, in run_checks
    self.run_ast_checks()
  File "/usr/local/lib/python3.6/site-packages/flake8/checker.py", line 474, in run_ast_checks
    row, column = self._extract_syntax_information(exception)
  File "/usr/local/lib/python3.6/site-packages/flake8/checker.py", line 441, in _extract_syntax_information
    if column > 0 and token and isinstance(exception, SyntaxError):
TypeError: '>' not supported between instances of 'str' and 'int'

Compilation exited abnormally with code 1 at Thu Mar  9 23:37:20

However, if execute the command flake8 --format\=default --stdin-display-name /Users/galeo/Desktop/flycheck-error.py - < /Users/galeo/Desktop/flycheck-error.py used by flycheck in the shell, it works OK!

Using pip3 install flake8 makes flake8 check the syntax for Python 3, here is the test code that could reproduce the error:

# flycheck-error.py
def allo(θ):
    print(θ)

This is the flycheck-verify-setup output:

Syntax checkers for buffer flycheck-error.py in python-mode:

  python-flake8
    - may enable:         yes
    - executable:         Found at /usr/local/bin/flake8
    - configuration file: Not found

  python-pylint
    - may enable:         Automatically disabled!
    - executable:         Not found
    - configuration file: Not found

  python-pycompile
    - may enable: yes
    - executable: Found at /Users/galeo/.virtualenvs/argo_env3/bin/python

Flycheck Mode is enabled. Use C-u C-c ! x to enable disabled checkers.

--------------------

Flycheck version: 31snapshot (package: 20170223.247)
Emacs version:    25.1.1
System:           x86_64-apple-darwin16.0.0
Window system:    ns

[LeeiFrankJaw]

I got the same problem. I can run flake8 from the shell without any problem but it throws error when used with flycheck. In fact, elpy utilize flake8 too but doesn't observe the problem.

[dekkers]

I had the same problem, flake8 gives the error because it doesn't get utf-8 as input. Doing C-h C RET my settings were:

Defaults for subprocess I/O:
  decoding: - -- undecided-unix (alias: unix)

  encoding: 1 -- iso-latin-1-unix (alias: iso-8859-1-unix latin-1-unix)

Adding (set-default-coding-systems 'utf-8) to .emacs that changed to:

Defaults for subprocess I/O:
  decoding: U -- utf-8-unix (alias: mule-utf-8-unix)

  encoding: U -- utf-8-unix (alias: mule-utf-8-unix)

Which results in emacs giving the buffer to flake8 as utf-8.

[fmdkdd]

@dekkers Good find! @LeeiFrankJaw @galeo, does changing the encoding in Emacs fixes this for you?

I'm not sure we should fix this directly in Flycheck. But, we could indicate in the documentation to set your coding system to utf-8. And/or to check it in a :verify for flake8.

What do you think @cpitclaudel ?

[cpitclaudel]

I'm not sure. I think we should be consistent about the encoding we use to send files over to checkers, especially through stdin.

[fmdkdd]

What about using the encoding for the buffer? That would probably be the less surprising behavior.

[cpitclaudel]

Possibly? Or we could standardize and always send utf-8

[fmdkdd]

That was my first thought as well, but then I know utf-8 is not universally used, at least for text. The question is: would it ever make sense to not send utf-8 to a checker? If it does, we would then have to introduce a variable to toggle this behavior, or worse, to set the encoding used by Flycheck, which seems redundant with the Emacs settings.

[galeo]

Hi, @dekkers @fmdkdd

The settings of mine:

Default coding system (for new files):
  U -- utf-8 (alias: mule-utf-8)

Coding system for keyboard input:
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for terminal output:
  U -- utf-8 (alias: mule-utf-8)

Coding system for inter-client cut and paste:
  U -- utf-8 (alias: mule-utf-8)

Defaults for subprocess I/O:
  decoding: U -- utf-8-unix (alias: mule-utf-8-unix)

  encoding: U -- utf-8-unix (alias: mule-utf-8-unix)

The coding system is always set to utf-8.

[cpitclaudel]

If it does, we would then have to introduce a variable to toggle this behavior, or worse, to set the encoding used by Flycheck, which seems redundant with the Emacs settings.

Right — if the checker accepts input from disk too, it should be fine to send it data in the file's original encoding. I wonder what (currently) happens if a user inserts an unsupported character in a buffer and flycheck runs right after that.

[fmdkdd]

@galeo Curious. If your buffer is in utf-8, and your coding system prefs are utf-8 everywhere, then I don't know what's happening here.

I've tried to reproduce with your example, but flake8 does not complain about the unicode character here. (Granted, I'm using 24.5 on Linux).

Maybe a package is overriding your coding system somehow? Have you tried with to reproduce from emacs -Q?

Syntax checkers for buffer flycheck-error.py in python-mode:

  python-flake8
    - may enable:         yes
    - executable:         Found at /usr/local/bin/flake8
    - configuration file: Not found

  python-pylint
    - may enable:         Automatically disabled!
    - executable:         Not found
    - configuration file: Not found

  python-pycompile
    - may enable: yes
    - executable: Found at /usr/bin/python

--------------------

Flycheck version: 31snapshot (package: 20170415.1006)
Emacs version:    24.5.1
System:           x86_64-pc-linux-gnu
Window system:    x

C-h C RET:

Coding system for saving this buffer:
  U -- utf-8-unix (alias: mule-utf-8-unix)

Default coding system (for new files):
  U -- utf-8 (alias: mule-utf-8)

Coding system for keyboard input:
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for terminal output:
  U -- utf-8-unix (alias: mule-utf-8-unix)

Coding system for inter-client cut and paste:
  nil
Defaults for subprocess I/O:
  decoding: U -- utf-8-unix (alias: mule-utf-8-unix)

  encoding: U -- utf-8-unix (alias: mule-utf-8-unix)

[galeo]

@fmdkdd I've made an investigation and could confirm that it works OK when start emacs from a shell. However, when start emacs from GUI, the problem occurs.

The environment variables inherited by emacs from a shell and GUI are different, I could not figure out what causes this problem.

I am using macOS Sierra 10.12.4 and emacs 25.2 installed with Homebrew (--with-cocoa).

[fmdkdd]

@galeo Ah! Thanks for looking into it :) If it's an environment issue, try to follow these instructions.

[wd]

I meet this problem for a long time, and found the way to fix it today.

I use fish as my default shell. And locale command output is:

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

The important setting here is LC_CTYPE, so first make sure this setting in your shell is correct.

Then, open emacs and run shell command use M-x shell-command input locale and check if the output LC_CTYPE is correct.

I fix this by create a file ~/.profile, and install exec-path-from-shell(I haven't check if this is necessary).

$ cat ~/.profile
export LANG='en_US.UTF-8'
export LC_CTYPE='UTF-8'

Hope this will help someone meet this problem too.

[galeo]

It works. Thanks so much!

[hzc989]

I meet this problem for a long time, and found the way to fix it today.

I use fish as my default shell. And locale command output is:

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=

The important setting here is LC_CTYPE, so first make sure this setting in your shell is correct.

Then, open emacs and run shell command use M-x shell-command input locale and check if the output LC_CTYPE is correct.

I fix this by create a file ~/.profile, and install exec-path-from-shell(I haven't check if this is necessary).

$ cat ~/.profile
export LANG='en_US.UTF-8'
export LC_CTYPE='UTF-8'

Hope this will help someone meet this problem too.

this one works for me.

[iburunat]

I fix this by create a file ~/.profile, and install exec-path-from-shell(I haven't check if this is necessary).

$ cat ~/.profile
export LANG='en_US.UTF-8'
export LC_CTYPE='UTF-8'

Hope this will help someone meet this problem too.

this one works for me.

Thanks for this. However, setting a .profile file does not alter my Emacs locale after restarting Emacs (all LC categories set to C currently), although my system's locale shows the right settings. I wonder what could be the problem.