Use python -c to invoke pylint and flake8 #1113

cpitclaudel · 2016-09-27T23:54:13Z

Use python -m to invoke pylint and flake8
Part of GH-1055.

cpitclaudel · 2016-09-28T00:42:46Z

LGTM :)

swsnr · 2016-09-28T08:35:27Z

@cpitclaudel I must admit that I'm not sure whether I'd like to merge this pull request. Can we discuss the problems that this is supposed to solve first, over at #1055?

swsnr · 2016-09-28T13:39:47Z

@cpitclaudel Independently of the discussion in #1055 we need custom :predicates for these syntax checkers now since we need to check whether the modules are actually available.

cpitclaudel · 2016-09-28T13:50:56Z

@cpitclaudel Independently of the discussion in #1055 we need custom :predicates for these syntax checkers now since we need to check whether the modules are actually available.

Ah, very good point. That is one downside of this PR. :predicate or :enabled?

swsnr · 2016-09-28T14:00:00Z

@cpitclaudel I'd go for :enabled. After all that's not something that'd change frequently would it?

Generally I think we should slowly move to use :enabled for anything that checks whether a checker is installed, as these are facts that scarcely change. We should leave :predicate for checks that really change frequently, like whether the buffer is saved, or whether it contains a certain text or so.

swsnr

Misses an :enabled that checks whether the corresponding modules are installed.

swsnr · 2016-09-28T14:01:04Z

LGTM, but I "requested changes" for the missing :enabled.

swsnr · 2016-10-27T09:54:26Z

@cpitclaudel Did you get a chance to work on this pull request meanwhile?

cpitclaudel · 2016-10-27T11:56:12Z

Barely. I'll try to get some time before the week-end

swsnr · 2016-10-27T12:51:40Z

@cpitclaudel No hurry, was just curious.

fmdkdd · 2017-07-17T16:00:25Z

@cpitclaudel Any update on this one?

cpitclaudel · 2018-01-28T18:19:17Z

I looked into this again, but there's a problem I didn't consider at the time: backwards compatibility. If people have custom paths for flake8 or pylint, this change will break them.

There's multiple ways to proceed.

Accept the breakage, document it, and move on; the problem is that the fix for users won't be entirely trivial (it's tricky to load a python module that's not in your Python path), unless they use a venv in which flake8 or pylint is installed (in which case they'll just have to point Flycheck to the venv's Python instead of the venv's pylint.
Make it possible to use either form: we add a new variable flycheck-flake8-python-executable, which if set takes precedence over flycheck-flake8-executable. This might be tricky to implement in today's Flycheck.
Same as above, but use a single variable for the executable, and add -m flake8 if we think the executable is python instead of a flake8 binary. We could either do the detection automatically (pattern-match for python(\d(\.\d)?)?(\.exe)?) or add a setting. If we do that, we can also change the default value of the checker executable to python, so we keep backwards compatibility while getting the benefits of python -m
Abandon this PR. I don't like that much, because I think it's harder than it should be ATM to mix python2 and python3 sources in Flycheck.

WDYT?

dakra · 2018-01-29T02:27:45Z

Is it possible to make python -m flake8 the default command and if someone customizes it, it'll just use that?

cpitclaudel · 2018-01-29T06:20:45Z

Is it possible to make python -m flake8 the default command and if someone customizes it, it'll just use that?

Not easily (at least I think, but I'd like to be proven wrong); Flycheck has a pretty deeply-embedded notion of checker executables, where each checker has it's own binary + arguments passed to it. python -m flake8 is an executable + 2 arguments, so what I'm proposing to do is to change the flake8 executable to "python" instead of flake8, and to extend the argument list with -m flake8; for compatibility, I'm also suggesting to make the argument list additions conditional, so that they only happen if the executable "looks like" python, rather than flake8.

ibizaman · 2018-02-01T06:29:17Z

Not especially pushing for this PR, but one use-case I have where calling pylint through python -m pylint is necessary is when having a .pylintrc like so:

[MASTER]

load-plugins=
    my_module.my_helper

This is the error I get in emacs:

Suspicious state from syntax checker python-pylint: Flycheck checker python-pylint returned non-zero exit code 1, but its output contained no errors: Using config file ./.pylintrc
Traceback (most recent call last):
  File ".virtualenvs/python2.7/bin/pylint", line 11, in <module>
    sys.exit(run_pylint())
  File ".virtualenvs/python2.7/lib/python2.7/site-packages/pylint/__init__.py", line 16, in run_pylint
    Run(sys.argv[1:])
  File ".virtualenvs/python2.7/lib/python2.7/site-packages/pylint/lint.py", line 1312, in __init__
    linter.load_plugin_modules(plugins)
  File ".virtualenvs/python2.7/lib/python2.7/site-packages/pylint/lint.py", line 494, in load_plugin_modules
    module = modutils.load_module_from_name(modname)
  File ".virtualenvs/python2.7/lib/python2.7/site-packages/astroid/modutils.py", line 190, in load_module_from_name
    return load_module_from_modpath(dotted_name.split('.'), path, use_sys)
  File ".virtualenvs/python2.7/lib/python2.7/site-packages/astroid/modutils.py", line 232, in load_module_from_modpath
    mp_file, mp_filename, mp_desc = imp.find_module(part, path)
ImportError: No module named my_module

Try installing a more recent version of python-pylint, and please open a bug report if the issue persists in the latest release.  Thanks!

I fixed that in another way, by adding:

  (with-eval-after-load 'flycheck
    (setq flycheck-command-wrapper-function
          (lambda (command)
            (if (null (string-match "pylint" (car command)))
                command
                (let* ((new-prefix (replace-regexp-in-string "pylint$" "python" (car command)))
                       (new-rest (append '("-m" "pylint") (cdr command)))
                       (new-command (append (list new-prefix) new-rest)))
                  new-command)))))

I also needed to set the default-directory to be the project root instead of the file's directory because other my_module wasn't always found. I did that in the .dir-locals.el with:

((nil . ((eval . (setq default-directory (locate-dominating-file default-directory ".dir-locals.el"))))))

EDIT: corrected the flycheck-command-wrapper-function snippet as it was a bit too aggressive.

cpitclaudel · 2018-02-01T13:53:51Z

Thank, that's an interesting data point. I think your solution is similar to mine.
How does your working directory setup looks like?

fmdkdd · 2018-02-10T15:55:15Z

I looked into this again, but there's a problem I didn't consider at the time: backwards compatibility. If people have custom paths for flake8 or pylint, this change will break them.

I'm not sure it's worth hanging onto backward compatibility on this one. python-pycompile is already using python -m. If users want to preserve the previous behavior, they can always override the checker with the previous version.

In this case, there is an even simpler trick:

(setf (flycheck-checker-get 'python-flake8 'command)
      (cddr (flycheck-checker-get 'python-flake8 'command))

would drop the python -m part of the command and restore the previous behavior.

fmdkdd · 2018-02-10T15:56:26Z

flycheck.el

@@ -8022,6 +8040,10 @@ Requires Flake8 3.0 or newer. See URL
            (id (one-or-more (any alpha)) (one-or-more digit)) " "
            (message (one-or-more not-newline))
            line-end))
+  :enabled (lambda () (and flycheck-flake8-python-executable


Is this a new variable that's not in this branch? Or did you mean flycheck-python-flake8-executable? The latter is already checked by Flycheck.

cpitclaudel · 2018-02-12T14:36:51Z

I updated the PR, but there's one more issue. From the docs of python -m:

If this option is given, the first element of sys.argv will be the full path to the module file. As with the -c option, the current directory will be added to the start of sys.path.

The addition of the current directory means that large projects with names that conflict with the standard library, like Sphinx, can't be checked:

python -m pylint -r n --output-format text --msg-template \{path\}\:\{line\}\:\{column\}\:\{C\}\:\{symbol\}\:\{msg\} --rcfile /home/clement/.emacs.d/external-config/.pylintrc /home/clement/.local/lib/python3.5/site-packages/sphinx/environment.py
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/clement/.local/lib/python2.7/site-packages/pylint/__main__.py", line 10, in <module>
    pylint.run_pylint()
  File "/home/clement/.local/lib/python2.7/site-packages/pylint/__init__.py", line 16, in run_pylint
    from pylint.lint import Run
  File "/home/clement/.local/lib/python2.7/site-packages/pylint/lint.py", line 57, in <module>
    import astroid
  File "/home/clement/.local/lib/python2.7/site-packages/astroid/__init__.py", line 54, in <module>
    from astroid.nodes import *
  File "/home/clement/.local/lib/python2.7/site-packages/astroid/nodes.py", line 54, in <module>
    from astroid.scoped_nodes import (
  File "/home/clement/.local/lib/python2.7/site-packages/astroid/scoped_nodes.py", line 25, in <module>
    import io
  File "io.py", line 12, in <module>
    from docutils.readers import standalone
  File "/usr/local/lib/python2.7/dist-packages/docutils/readers/standalone.py", line 13, in <module>
    from docutils import frontend, readers
  File "/usr/local/lib/python2.7/dist-packages/docutils/frontend.py", line 38, in <module>
    import optparse
  File "/usr/lib/python2.7/optparse.py", line 419, in <module>
    _builtin_cvt = { "int" : (_parse_int, _("integer")),
  File "/usr/lib/python2.7/gettext.py", line 584, in gettext
    return dgettext(_current_domain, message)
  File "/usr/lib/python2.7/gettext.py", line 548, in dgettext
    codeset=_localecodesets.get(domain))
  File "/usr/lib/python2.7/gettext.py", line 483, in translation
    mofiles = find(domain, localedir, languages, all=1)
  File "/usr/lib/python2.7/gettext.py", line 440, in find
    for nelang in _expand_lang(lang):
  File "/usr/lib/python2.7/gettext.py", line 133, in _expand_lang
    from locale import normalize
ImportError: cannot import name normalize

(this happens because Sphinx contains a "locale" module.

cpitclaudel · 2018-02-12T15:46:23Z

Fixed by moving to python -c. Let me know what you think.

fmdkdd

LGTM. I'm not convinced of the need for backward-compatibility on this one, but I don't write enough Python, so I'll trust your judgment.

This can go into CHANGES.rst Improvements section.

fmdkdd · 2018-02-16T13:49:36Z

flycheck.el

@@ -8764,7 +8803,8 @@ Update the error level of ERR according to

 Requires Flake8 3.0 or newer. See URL
 `https://flake8.readthedocs.io/'."
-  :command ("flake8"


Please add a comment saying why we don't call flake8 directly and linking to this PR (or #1055).

fmdkdd · 2018-02-16T13:51:59Z

flycheck.el

+  "Determines whether CHECKER needs a `-m module-name' argument.
+Previous versions of Flycheck called pylint and flake8 directly;
+this check ensures that we don't break existing code."
+  (not (string-match-p (concat "\\(pylint\\|flake8\\)"


Could you use rx to make the regexp more readable?

Sure. Sounds like the perfect time to advertise https://github.com/cpitclaudel/easy-escape, too :)

fmdkdd · 2018-02-16T14:17:34Z

flycheck.el

+           (string-trim (buffer-string))))))
+
+(defun flycheck-python-needs-module-p (checker)
+  "Determines whether CHECKER needs a `-m module-name' argument.


I guess it's not really -m module-name, since we end up using -c instead. But the intent is the same?

Thanks, I changed the docstring.

Part of GH-1055. Ideally we'd use 'python -m ...' instead of 'python -c ...', but Python adds the current directory to sys.path in -c and -m mode, which confuses pylint (for example, 'python -m pylint' breaks when Flychecking Sphinx, whose sources include a 'locale' directory that conflicts with Python's standard 'locale' module).

cpitclaudel · 2018-02-17T14:11:31Z

This can go into CHANGES.rst Improvements section.

Yup, I already added it :)

cpitclaudel · 2018-02-17T15:34:07Z

Question: do we actually care about argument order in docstrings? I'm temper to turn checkdoc-arguments-in-order-flag off.

fmdkdd · 2018-02-17T15:37:52Z

I don't mind about the order no.

cpitclaudel · 2018-02-17T16:42:09Z

OK, all green.

swsnr added status: blocked component: checkers labels Sep 28, 2016

swsnr mentioned this pull request Sep 28, 2016

Use the python-shell-interpreter command for Python checkers #1055

Closed

swsnr suggested changes Sep 28, 2016

View reviewed changes

swsnr added status: pending and removed status: blocked labels Oct 10, 2016

cpitclaudel force-pushed the 1055-python-executable branch from 4f4b375 to 9185eb3 Compare January 28, 2018 18:08

fmdkdd requested changes Feb 10, 2018

View reviewed changes

cpitclaudel force-pushed the 1055-python-executable branch from 9185eb3 to 3971a82 Compare February 12, 2018 14:34

cpitclaudel force-pushed the 1055-python-executable branch from 3971a82 to 7c8ce34 Compare February 12, 2018 15:45

cpitclaudel changed the title ~~Use python -m to invoke pylint and flake8~~ Use python -c to invoke pylint and flake8 Feb 12, 2018

cpitclaudel force-pushed the 1055-python-executable branch from 7c8ce34 to 71b2c7a Compare February 12, 2018 16:23

cpitclaudel force-pushed the 1055-python-executable branch 7 times, most recently from 2181b02 to 8154437 Compare February 12, 2018 16:36

fmdkdd approved these changes Feb 16, 2018

View reviewed changes

cpitclaudel force-pushed the 1055-python-executable branch from 8154437 to 6e45989 Compare February 17, 2018 14:09

Don't check for argument order in docstrings

1100f1d

cpitclaudel force-pushed the 1055-python-executable branch from 61fbf04 to 1100f1d Compare February 17, 2018 16:28

fmdkdd merged commit 1ec9397 into master Feb 17, 2018

fmdkdd deleted the 1055-python-executable branch February 17, 2018 20:01

fmdkdd mentioned this pull request Jun 5, 2018

Add mypy as a flycheck checker #1354

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use python -c to invoke pylint and flake8 #1113

Use python -c to invoke pylint and flake8 #1113

cpitclaudel commented Sep 27, 2016

cpitclaudel commented Sep 28, 2016

swsnr commented Sep 28, 2016

swsnr commented Sep 28, 2016

cpitclaudel commented Sep 28, 2016

swsnr commented Sep 28, 2016

swsnr left a comment

swsnr commented Sep 28, 2016 •

edited

Loading

swsnr commented Oct 27, 2016

cpitclaudel commented Oct 27, 2016

swsnr commented Oct 27, 2016

fmdkdd commented Jul 17, 2017

cpitclaudel commented Jan 28, 2018

dakra commented Jan 29, 2018

cpitclaudel commented Jan 29, 2018

ibizaman commented Feb 1, 2018 •

edited

Loading

cpitclaudel commented Feb 1, 2018

fmdkdd commented Feb 10, 2018

fmdkdd Feb 10, 2018

cpitclaudel commented Feb 12, 2018

cpitclaudel commented Feb 12, 2018

fmdkdd left a comment

fmdkdd Feb 16, 2018

cpitclaudel Feb 17, 2018

fmdkdd Feb 16, 2018

cpitclaudel Feb 17, 2018

fmdkdd Feb 16, 2018

cpitclaudel Feb 17, 2018

cpitclaudel commented Feb 17, 2018

cpitclaudel commented Feb 17, 2018

fmdkdd commented Feb 17, 2018

cpitclaudel commented Feb 17, 2018

Use python -c to invoke pylint and flake8 #1113

Use python -c to invoke pylint and flake8 #1113

Conversation

cpitclaudel commented Sep 27, 2016

cpitclaudel commented Sep 28, 2016

swsnr commented Sep 28, 2016

swsnr commented Sep 28, 2016

cpitclaudel commented Sep 28, 2016

swsnr commented Sep 28, 2016

swsnr left a comment

Choose a reason for hiding this comment

swsnr commented Sep 28, 2016 • edited Loading

swsnr commented Oct 27, 2016

cpitclaudel commented Oct 27, 2016

swsnr commented Oct 27, 2016

fmdkdd commented Jul 17, 2017

cpitclaudel commented Jan 28, 2018

dakra commented Jan 29, 2018

cpitclaudel commented Jan 29, 2018

ibizaman commented Feb 1, 2018 • edited Loading

cpitclaudel commented Feb 1, 2018

fmdkdd commented Feb 10, 2018

fmdkdd Feb 10, 2018

Choose a reason for hiding this comment

cpitclaudel commented Feb 12, 2018

cpitclaudel commented Feb 12, 2018

fmdkdd left a comment

Choose a reason for hiding this comment

fmdkdd Feb 16, 2018

Choose a reason for hiding this comment

cpitclaudel Feb 17, 2018

Choose a reason for hiding this comment

fmdkdd Feb 16, 2018

Choose a reason for hiding this comment

cpitclaudel Feb 17, 2018

Choose a reason for hiding this comment

fmdkdd Feb 16, 2018

Choose a reason for hiding this comment

cpitclaudel Feb 17, 2018

Choose a reason for hiding this comment

cpitclaudel commented Feb 17, 2018

cpitclaudel commented Feb 17, 2018

fmdkdd commented Feb 17, 2018

cpitclaudel commented Feb 17, 2018

swsnr commented Sep 28, 2016 •

edited

Loading

ibizaman commented Feb 1, 2018 •

edited

Loading