Spyder very slow when working on large python files

[impact27]

Description

What steps will reproduce the problem?

A few things in spyder are slowing down spyder considerably, mostly whrn working with big files. The functions that take most of the time are:

• the paintEvent from spyder/plugins/editor/panels/scrollflag.py
  • Mostly computing the flags and adding all the painters
• __mark_occurences in spyder/plugins/editor/widgets/codeeditor.py

Versions

• Spyder version: 5.0.0.dev0 acb3dbf
• Python version: 3.7.5
• Qt version: 5.12.6
• PyQt5 version: 5.12.3
• Operating System: Darwin 19.0.0

Dependencies

cloudpickle >=0.5.0          :  1.2.2 (OK)
pygments >=2.0               :  2.4.2 (OK)
qtconsole >=4.6.0            :  4.7.0.dev0 (OK)
nbconvert >=4.0              :  5.6.1 (OK)
sphinx >=0.6.6               :  2.2.1 (OK)
pylint >=0.25                :  2.4.4 (OK)
psutil >=0.3                 :  5.6.5 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  18.1.1 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.9.1 (OK)
spyder_kernels >=1.8.1;<2.0.0:  1.9.0.dev0 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
intervaltree                 :  None (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
sympy >=0.7.3                :  1.4 (OK)
cython >=0.21                :  None (NOK)
IPython >=4.0                :  7.11.0.dev (OK)
matplotlib >=2.0.0           :  3.1.2 (OK)
pandas >=0.13.1              :  0.25.3 (OK)
numpy >=1.7                  :  1.17.4 (OK)
scipy >=0.17.0               :  1.3.3 (OK)
pyls >=0.31.2;<0.32.0        :  0.31.1 (NOK)
applaunchservices >=0.1.7    :  0.2.1 (OK)

[impact27]

An example: (rename to .py)

pdc_lib copy.py.txt

[ccordoba12]

Could you help us with this one @impact27?

[jnsebgosselin]

I think I can help with optimizing the scrollflag paintEvent if you need help with it.

[ccordoba12]

That'd be really appreciated, thanks!

[jnsebgosselin]

That'd be really appreciated, thanks!

Ok I'm on it. I won't touch the __mark_occurences in spyder/plugins/editor/widgets/codeeditor.py though.

[bcolsen]

This would be related to #10992

[impact27]

I opened #11036 for __mark_occurences
I go from taking 33s for __mark_occurences to 1.2s (mostly taken by __find_next)
Also process_code_analysis goes from 3.7 to 0.2s