Continuously increasing memory usage when pylint is run via its API #792
Description
Steps to reproduce
When using pylint in-process to lint a file (in this case in vim), the memory usage of the editor is continually increasing. The increase is even higher when the cache is cleared. I believe that this is caused because the module is still referenced by lru_cache even though it has been deleted from the manager cache and each new parsing adds a new cache entry.
import gc
import astroid
import psutil
from pylint import lint
from pylint.reporters.base_reporter import BaseReporter
class Reporter(BaseReporter):
def _display(self, layout):
pass
def run1():
print('run without clearing cache')
for _ in range(0, 5):
lint.Run(['--reports', 'n', 'pyparsing1.py'], reporter=Reporter(), exit=False)
print(psutil.Process().memory_info().rss)
def run2():
print('run with cache clear')
for _ in range(0, 5):
lint.Run(['--reports', 'n', 'pyparsing1.py'], reporter=Reporter(), exit=False)
astroid.builder.MANAGER.astroid_cache.clear()
gc.collect()
print(psutil.Process().memory_info().rss)
def run3():
print('run with clearning only checked module cache')
for _ in range(0, 5):
lint.Run(['--reports', 'n', 'pyparsing1.py'], reporter=Reporter(), exit=False)
del astroid.builder.MANAGER.astroid_cache['pyparsing1']
gc.collect()
print(psutil.Process().memory_info().rss)
def main():
run1()
run2()
run3()
if __name__ == '__main__':
main()The linted file is 253KB pyparsing.py renamed to pyparsing1.py.
run without clearing cache
118296576
130756608
143630336
156053504
168882176
run with cache clear
181768192
263176192
345382912
427147264
507686912
run with clearning only checked module cache
589205504
615079936
641482752
665550848
689803264
Current behavior
Increasing memory usage
Expected behavior
Stable memory usage
python -c "from astroid import __pkginfo__; print(__pkginfo__.version)" output
2.4.1