Add an option to generate progress output

[chrisspen]

This is a feature request.

I'm currently using Pylint to check a very large codebase as part of a continuous build process. It now takes over 20 minutes to run, which isn't itself a problem. The problem is that Pylint doesn't output any progress information as it processes without encountering errors, so my build tool is failing Pylint because it thinks it's timing out.

Obviously, I can increase the timeout used by my tool, but I'll have to continually maintain this as my code grows. I don't want to outright disable the timeout check for cases where, due to low memory or other strange edge-cases, Pylint genuinely hangs and needs to be forcefully terminated. The ideal solution would be if Pylint could output some sort of progress, such as the names of files scanned, or better yet, an overall percent progress until completion.

I'm currently using 1.7.1, and --help doesn't appear to indicate there's currently any feature like this.

[rogalski]

Yes, we do not have this kind of feature right now.

For very simple solution you may try patching pylint yourself by modifying pylint.lint.PyLinter#_do_check with something like:

       for descr in self.expand_files(files_or_modules):
           modname, filepath, is_arg = descr['name'], descr['path'], descr['isarg']
           if not self.should_analyze_file(modname, filepath, is_argument=is_arg):
               continue
           print("Checking module", modname)  # added line
           self.set_current_module(modname, filepath)

This will work for simple non-parallel builds.

In general, to have some form of progress bar built-in in pylint, we'll need to make sure both sequential and parallel builds will work fine. I'd prefer to discuss more about requirements and specification before jumping to implementation.

[Pierre-Sassoulas]

I think a proposal or a proof of concept would greatly accelerate the process. Maintainers are busy running in order to stand still (new python interpreters, open pull request, upgrade of dependencies...), so unless someone interested in an issue push it, those kind of big issues are not going to be solved.

[MalanB]

Hi Folks, I was working on pandas repo for a PR and wanted to run pylint. It's a pretty big repository and Pylint runs for very long time without showing any progress.

I would like to add a progressbar or STDOUT the file being checked in Pylint. Do you have any concerns over it?

[Pierre-Sassoulas]

This is a high priority issue so any help is appreciated 😄

In order to be able to handle parallel builds a specification still needs to be made. We have some issues with parallelization however, so maybe other maintainers would agree to not handle parallel builds until #2525 is fixed ?

An option also need to be defined so it's possible to remove this output. I don't want to add a dependency to tqdm for all install (we had a lot of problem with dill not being available for python 3.11 for the longest time, I don't want to increase the probability of this kind of issue happening more often). So if it makes sense to add a dependency it should be optional like py-enchant.

[MalanB]

Thank you. I will wait till a spec has been defined and agreed by all maintainers.

[Pierre-Sassoulas]

A person pushing for this feature with a design proposal or a proof of concept would accelerate the implementation a lot. Maintainers are all busy running in order to stand still (new interpreters, upgrading dependencies, reviewing merge requests, triaging issues, releasing, fixing crash and bugs, etc.). As you probably saw I only reacted to your comment even if the issue is high priority :)