Problems with sip 4.14.3 and PyQt 4.9.6 (was: Slow parsing in editor window) #100
Comments
|
Did the slowness start right after updating libraries on your system? |
|
Well I'm using openSuse Tumbleweed (a "rolling release") so some libraries may have been updated since the last time i used Frescobaldi without problems (I did not use it for a couple of weeks). An updated library may be the cause of the problem because exactly the same program that used to work flawlessly before started giving those problems when I wanted to use it again about a week ago. Recompiling the stable 2.0.8 or the git version did not help at all. After rereading what I just wrote I think you may be right about some problems with the system libraries. However do you have any idea what is the best approach to solve this? Is Frescobaldi maybe incompatible with certain versions of the libraries it used or should I maybe try to create a "statically linked" version or so? |
|
What happens if you close the PDF view, re-save the ly file and restart Frescobaldi so that the PDF view does not show (and therefore does not load)? This looks like a problem with different PyQt libraries being loaded at the same time because e.g. popplerqt4 might link to the wrong SIP or PyQt libraries. |
|
When I follow the steps you describe I keep the same behavior as before. I did notice however that Frescobaldi takes 100% CPU during the "wait" times, so at starting up and for example when adding a "%" at the beginning of a line to comment it out. So effectively it is not stalling but just very busy at those moments Also tried to "downgrade" the SIP libraries, but then "Python-Poppler-Qt4" didn't build anymore due to "a wrong version of the implemented sip API". Therefore I upgraded the SIP libraries again. Is there a way to tell whether different PyQt libraries are being loaded at the same time? (According to the "package manager" there is only a single PyQt installed......) |
|
Ok, we can rule our the library/dependency problem per se. But you made an interesting remark about Frescobaldi working well when all of LilyPond input is on one (or a few) lines. Many code in Frescobaldi iterates over text lines (QTextBlock) instances. E.g. the syntax highlighting/parsing (even if turned off in the View menu), auto-indenting, matching parenthesis highlighting etc etc. There may be a difference in how QTextBlocks are handled by Python in more recent SIP versions. That might cause interesting problems. I'll try to look into it. |
|
What happens if you switch off the automatic indent? |
|
Turning "Automatic indent" off does not make a difference. However turning "Auto-completion" (I hope the translation from "Automatisch aanvullen" is correct) does make a big difference!! When that option is turned off, the editor is fast like it was before (or at least feels like that) When compiling the "Lilypond-output" window is still not updated until the compilation process is finished. At first it seems the compilation itself is also much slower than it used to be, and also slower than on a different much older and slower computer. Because Lilypond is an external program that is executed from by Frescobaldi I ran a number of compilations while simultaneously looking at the CPU usage of the Frescobaldi and the Lilypond processes. It turned out that the Lilypond process existed for a relatively short period of time and used a relatively small amount of CPU (<40%). Before and after Lilypond is executed the Frescobaldi process takes approximately 100% CPU during a longer period of time. (say Frescobaldi 45% of time before, then Lilypond 10% of time, Frescobaldi 45% of time after). This makes me think there may be a delay in Frescobaldi before and after the actual call to Lilypond is made..but as I'm not familiar with the Frescobaldi internals this is pure guessing... |
|
Ok, the autocompletion stuff can be slow and time consuming because it searches files that are \include'd for variable definitions. And it is not yet threaded, meaning that reading a file shortly blocks the GUI. Do you use \include statements? Also when running LilyPond, Frescobaldi searches included files for commands that might alter the name of the generated output file(s). Does Frescobaldi also block the GUI and use lots of CPU before and after running LilyPond with auto-complete turned off? |
|
Ok, that makes sense because triggering the auto-complete by hitting Ctrl+Space does still have the delay. Weird that even on the older/slower computer the search seems to be pretty fast (as is the autocompletion) and it used to be fast on this computer as well... Frescobaldi does block the GUI and uses lots of CPU before/after running Lilypond with auto-complete turned off. It also just caught my attention that with auto-complete turned off, editing works fine after the file content is shown, but the delay/blocking is still present when opening a ly file. Once the file is opened and shown this delay is gone. |
|
Thanks for all the effort you take in helping me to locate the problem. I'll now try to look in what could cause the delays. Something in autocomplete and something that's done on first load and before LilyPond is run. Maybe, if it's possible, I'd like to have a LilyPond file that demonstrates the problem on your machine. |
|
That should not be any problem at all. Should I past some file content here? (it seems only images can be attached directly) |
|
yeah that could be nice. I'm right now updating one of my systems to newer library versions. |
|
OK. As things happen with every ly file I just created a rather simple example which shows the mentioned behavior. In case some of the extra layout settings influence the overall behavior I did include the layout settings I normally use (basically they adjust the output size of the music to fit on a marching band size paper while still printing on A4 paper). Hopefully this does not complicate the example too much, otherwise let me know and I'll skip the \paper and the \layout parts for the example or try to create a better example! **** file begin **** % small test file to test the usage of frescobaldi
% effectively using rather adjusted \header \paper sections etc in order
% to have the output suitable for a marching-band sized paper (while printing on A4)
\version "2.12.0"
#(set-default-paper-size "a4")
#(set-global-staff-size 18)
\header {
% use the \markup construct to increase the font size if needed
title = \markup {\fontsize #+4 "Title"}
subtitle = ##f
subsubtitle = ##f
instrument = ##f
arranger = ##f
opus = ##f
meter = ##f
poet = ##f
tagline = ##F % the fesult is "Music engraving by Lilypond....."
composer = ##F % \markup {\fontsize #+2 "Comp: Trad."} %##f
piece = \markup {\fontsize #+1 "Trompet B" \normalsize \flat }
copyright = ##f
}
\paper {
% paper-height = 29.7\cm
% paper-width = 21.0\cm
page-top-space = 0\mm
page-spacing-weight = #10 % relative importance of vert-hor spacing, default 10
top-margin = 1.0\cm
bottom-margin = 15.7\cm %7\cm
left-margin = 1.0\cm
right-margin = 3.5\cm % defined but not working correctly according to the manual => use line-width
line-width = 16.5\cm
head-separation = 0\mm
foot-separation = 0\mm
after-title-space = 0\mm
before-title-space = 0\mm
between-title-space = 0\mm
indent = 0.0\cm
% vertical spacing of the systems
between-system-padding = #0.525
between-system-space = 1.6\cm
ragged-last-bottom = ##t %% set to ##t if your score is less than one page
ragged-bottom = ##f
page-count = #1
%system-count = #9 % set the number of systems in the output
% annotate-spacing = ##t % debug the spacings
% JUST SOME HEADER FOR A 2ND PAGE
evenHeaderMarkup = \markup \fill-line {
\left-column { \fromproperty #'header:piece }
\center-column {\fontsize #-4 \fromproperty #'header:title }
\right-column {\fromproperty #'page:page-number-string }%"/" \fromproperty #'page:page-count
% \on-the-fly #print-page-number-check-first \fromproperty #'page:page-number-string
% " "
% \fromproperty #'header:title
}
}
% layout block to influence the horizontal spacing things
\layout {
\context {
\Score
\override SpacingSpanner
#'base-shortest-duration = #(ly:make-moment 1 6) % larger values of the last number result in larger music!
\override SpacingSpanner
#'common-shortest-duration = #(ly:make-moment 1 6) % larger values of the last number result in larger music!
}
}
AvoiceAA = \relative c'{
\set Staff.instrumentName = #""
\set Staff.shortInstrumentName = #""
\clef treble
%staffkeysig
\key c \major
\clef treble
%bartimesig:
\time 4/4
c4 \mf d e f |
g a b c~ |
c1 \fermata |
c4 b a g |
f e d c~ |
c1 \fermata \bar"||"
c4 \mf d e f |
g a b c~ |
c1 \fermata |
c4 b a g |
f e d c~ |
c1 \fermata \bar"||"
c4 \mf d e f |
g a b c~ |
c1 \fermata |
c4 b a g |
f e d c~ |
c1 \fermata \bar"||"
c4 \mf d e f |
g a b c~ |
c1 \fermata |
c4 b a g |
f e d c~ |
c1 \fermata \bar"||"
c4 \mf d e f |
g a b c~ |
c1 \fermata |
c4 b a g |
f e d c~ |
c1 \fermata \bar"||"
c4 \mf d e f |
g a b c~ |
c1 \fermata |
c4 b a g |
f e d c~ |
c1 \fermata \bar"|."
}% end of last bar in partorvoice
\score {
<<
\context Staff = ApartA <<
\context Voice = AvoiceAA \AvoiceAA
>>
\override MultiMeasureRest #'expand-limit = #1 % always expand multi measure rests!
\set Score.skipBars = ##t
\set Score.melismaBusyProperties = #'()
\set Score.skipBars = ##t
\set countPercentRepeats = ##t
% \override Score.BarNumber #'break-visibility = #end-of-line-invisible %%every bar is numbered.!!!
% \set Score.barNumberVisibility = #all-bar-numbers-visible % also show the first bar number
% \set Score.barNumberVisibility = #(every-nth-bar-number-visible 5) % every nth barnumber is shown
\override Score.BarNumber #'self-alignment-X = #LEFT %#CENTER % center the bar numbers
#(set-accidental-style 'modern-cautionary)
% \set Score.markFormatter = #format-mark-box-letters %%boxed rehearsal-marks
\set Score.markFormatter = #format-mark-box-numbers %%boxed rehearsal-marks
% \set Score.markFormatter = #(lambda (mark context) (make-circle-markup (format-mark-numbers mark context)))
% \set Score.markFormatter = #(lambda (mark context) (make-circle-markup (format-mark-letters mark context)))
% \set Score.markFormatter = #format-mark-box-alphabet %%boxed rehearsal-marks including I
% \override Score.TimeSignature #'style = #'() %%makes timesigs always numerical
%% remove previous line to get cut-time/alla breve or common time
>>
}
**** file end ***** |
|
I am now using Frescobaldi: 2.0.9 which is in ubuntu 12.10, with no problems at all. Your libraries are even newer, so there might still be an issue with the new libraries which I'm not yet aware of. (But reading the change logs for SIP, Qt4 and PyQt4 I can't find anything that could explain the problems you're experiencing). |
|
I tried at a different computer with a recent openSUSE installed and did some experimenting. This other computer was running a 32-bit openSUSE (i586) whereas the first computer was running a 64-bit version. After downgrading some libraries I tried again, this time with |
|
Just tried downgrading on the first computer (64-bit) as well. Downgrading SIP lead to a downgrade of PyQt as well. |
|
Great. Well I got another report from someone having problems with the latest SIP/PyQt4 versions when using the Score Wizard. As soon as I hit those versions on one of my systems, I'll be able to look into it. |
|
Great. And many thanks for all your efforts already! |
|
Similar problem with archlinux (64-bit) Frescobaldi worked fine 1 month ago. After some upgrades it takes 30 secondes or more to open a document. When I try to set the fold/unfold option everything seems frozen (several seconds to display a menu). Downgrading to python2-sip-4.14.2-1 solves the problem |
|
Ok thanks. So sip-4.14.3 introduces the problems. I tried manually building sip-4.14.3 with PyQt4 but I get lots of segfaults...so I'll have to wait til my xubuntu introduces it before I can see what really happens and if that can be solved on the Frescobaldi side. It certainly looks like an unintended behaviour change in sip. |
|
Same critical issue here using Arch Linux Frescobaldi: 2.0.8 |
|
Same setup as pilmusic. I have run a call graph of the issue (http://imgh.us/open2.svg). This is how I made the callgraph: From the call graph, we can see that much time is spent in I hope this can be of use. Is there a way to completely remove highlighting? Something like a command-line flag, or a simple line that can be commented / uncommented... Warning: The SVG file is a bit large (544 kB), although Chromium here can open it fairly easily. |
|
Many thanks for the callgraph! I will try to make one myself, on my correctly functioning system, but I think I already see the problem. The problem is that all calls to tokeniter.tokens() result in a regeneration of the highlighting for that textblock, while the tokens should be cached in the QTextBlock().userData() objects. These cached tokens seem to be lost when using SIP 4.14.3 and PyQt 4.9.6, causing the lexer to run thousands of times, instead of just once! This looks like a bug in the PyQt 4.9.6 / sip 4.14.3 combo. Maybe it can be circumvented. Unfortunately there is (currently) no way to turn the parser off, even with highlighting turned off, the parser runs. |
|
Who is able to debug this somewhat further? Go to a python shell: then open one document (that contains some text on the first line) and be sure to have the document in the display. This should print a tuple of tokens, e.g.
If this raises an AttributeError, then we know the tokens are forgotten each time. |
|
Well, I found the difference in Pyqt4/Sip behaviour. When you set an instance attribute to a QTextBlockUserData instance after having the QTextBlockUserData instance assigned to a QTextBlock with QTextBlock.setUserData(), the python copy of QTextBlockUserData is recreated (forgetting the python instance attribute) when the userdata is requested again later on a new instance of the "same" QTextBlock (QTextBlock instances are recreated always). In sip 4.13.2 (and maybe even 4.14.2), sip would recognize the userdata object and the python instance attribute is not forgotten. I have posted a question regarding this on the PyQt mailing list. |
|
I have run your commands and I get this output: |
|
Yes, I also tested with a manually built sip/pyqt4 combo. This is the posting to the PyQt list. http://www.riverbankcomputing.com/pipermail/pyqt/2013-March/032457.html SIP apparently doesn't support anymore python instance attributes on wrapped C++ types, at least not of the QTextBlockUserData type (not a subclass of QObject). In sip-4.14.3, the C++ objects are rewrapped on each request and thus they forget python instance attributes. Whether this is a SIP bug or an intentional change, the problem can be circumvented in a satisfying way. I can derive from QTextBlockUserData and then set the tokens attribute to the instance. But there are some more places in Frescobaldi where this technique is used. |
|
Funny to see the tokenizer (State.tokens()) in the callgraph being called 500000+ times :-) It should only run once for each text line in a document.... |
|
Confirmation of the PyQt4 author that this is a PyQt4 bug. The upcoming nightly snapshot contains the fix! |
|
I tried to work around the bug, but it is not possible, unfortunately. Even when deriving from QTextBlockUserData, the Python wrapper of the C++ object is garbage collected and lost. It works for a few seconds but then the tokens are lost. I also see other PyQt4-based IDE's suffering from this problem (e.g. Ninja). I could simply disable the parsing but that would destroy almost all features like syntax highlighting, autocomplete, coloring matching braces, automatic indent, but also the music functions transpose, translate, etc etc. I could manually run the parser for such important music functions, but also in that case the tokens are expected to be cached, as some functions iterate backwards over the tokens (e.g. checking whether we are in a \relative music or not) and that will break as well when the tokens are forgotten each time. So I think I must state that Frescobaldi cannot be used with the SIP-4-14.3 / PyQt4-4.9.6 combo due to this bug. |
|
Finally I was able to make a workaround. When the highlighter is run, it saves state for everyblock using the userState() integer value. So we use that state and only parse the tokens for that line of text. It is still much slower than caching the tokens, but at least the highlighter does not re-run on the whole document on every request. Frescobaldi now remains usable I hope. Please test current git! |
|
Frescobaldi: 2.0.9 Python: 2.7.3 -- Qt: 4.8.4 -- PyQt4: 4.10 -- sip: 4.14.4 Traceback (most recent call last): |
|
ah thanks! actually that was caused by commit 5ecb5b3, fixed! |
|
Thanks a million wbsoft! Frescobaldi is working again! Let's get on with the music shall we :-) |
|
Great! Please test everything thouroughtly, as we're approaching the 2.0.9 release :-) Thanks to all for helping to alleviate this issue! |
|
Too bad I posted my report on GCode instead, it would have saved you some search! Thanks a lot! |
Frescobaldi used to work fast! However since some time the parsing in the editor window seems to be so slowwww that it renders Frescobaldi nearly useless...
This is what happens:
This happens for both the 2.0.8 release as the current git version and with the following versions of the used libraries:
Python: 2.7.3
Qt: 4.8.4
PyQt4: 4.9.6
sip: 4.14.3
Are these library versions not supported or am I overseeing some other things?
The text was updated successfully, but these errors were encountered: