High CPU / input lag - performance issues with 1.5.0-eap3-17-gd29f45b?

[gnl]

Can't really narrow it down much further than this, but since the upgrade to 1.5.0-eap3 I've been experiencing some performance issues often severely affecting input lag - I'd be happy to provide additional data / troubleshoot if required.

[thheller]

I upgraded to EAP4 recently (no EAP before) and performance has definitely taken a big hit. I can easily max out my CPU by just coding normally in a file.

These files are very bad:
https://github.com/thheller/shadow-build/blob/master/src/main/shadow/cljs/build.clj
https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/analyzer.cljc
https://github.com/clojure/clojurescript/blob/master/src/main/clojure/cljs/compiler.cljc

They are all pretty big files but the input lag wasn't this bad before.

I did a bit of sample profiling with JVisualVM and this looks suspicious:

You can get the full JVisualVM report here:
http://zilence.net/cursive/idea-hotspot.nps

I have no nREPL running so it must be something from Cursive.

[cursive-ide]

@thheller Does File->Invalidate caches and restart... help that? It looks like a lot of the time is in the BaseDataReader/Writer, which are the indexing classes.

[thheller]

@cursive-ide no it does not make a difference. After invalidating and waiting for the re-index to settle down I get the same CPU-heavy behaviour. The input lag does seem to get worse over time though.

I did a bunch of cleanup/refactor work recently which involved moving a fewdefn around, initially this works ok but after a while I have so sometimes wait a few seconds for Idea to recover before I can type again.

[cursive-ide]

Could either (or both) of you capture a CPU snapshot as detailed here? I'll take a look and see if I can see anything.

Also, if you look at the memory indicator (turned on using Settings→Appearance & Behavior→Appearance→Show memory indicator) do you see a lot of memory churn?

[thheller]

I have

# custom IntelliJ IDEA VM options

-Xms128m
-Xmx750m
-XX:ReservedCodeCacheSize=240m
-XX:+UseCompressedOops
-agentlib:yjpagent=delay=10000,probe_disable=*

in ~/Library/Preferences/IdeaIC2017.1/idea.vmoptions but I cannot start IDEA anymore.

All I get is when trying to launch via open, everything else just fails without any msg:

LSOpenURLsWithRole() failed with error -10810 for the file /Applications/IntelliJ IDEA CE.app.

[thheller]

2017-04-10 10:04:23.610 idea[22295:21066848] allVms required 1.8*,1.8+
2017-04-10 10:04:23.613 idea[22295:21066857] fullFileName is: /Applications/IntelliJ IDEA CE.app/Contents/bin/idea.vmoptions
2017-04-10 10:04:23.613 idea[22295:21066857] fullFileName exists: /Applications/IntelliJ IDEA CE.app/Contents/bin/idea.vmoptions
2017-04-10 10:04:23.613 idea[22295:21066857] Value of IDEA_VM_OPTIONS is (null)
2017-04-10 10:04:23.613 idea[22295:21066857] Processing VMOptions file at /Users/zilence/Library/Preferences/IdeaIC2017.1/idea.vmoptions
2017-04-10 10:04:23.613 idea[22295:21066857] Done
Error occurred during initialization of VM
Could not find agent library yjpagent on the library path, with error: dlopen(libyjpagent.dylib, 1): image not found```

[cursive-ide]

Assuming you're on the community edition, there's a section in the doc about that:

IntelliJ IDEA and PyCharm Community Editions do not include a YourKit profiler agent, because the YourKit agent library is not open-source. To take a CPU or memory snapshot with the Community Edition, you can download an evaluation version of YourKit Java Profiler, copy the yjpagent library to the IDE bin directory, add it to the .vmoptions as described above, and connect to the running IDE instance from the standalone YourKit Profiler UI.

Sorry, it's a bit more messing around. The other option would be to download an evaluation of IntelliJ Ultimate edition and use that to produce the dump. I'll try to reproduce it here too.

[thheller]

Maybe this helps: http://zilence.net/cursive/cursive-flight-recorder.jfr

It is a recording for Java Mission Control (jmc) which is slightly better than the JVisualVM recording. Switched to the Oracle JDK for this which made IDEA pretty ugly. Didn't do a whole lot of actual work with this. The recording is from me just typing random stuff into the source files at different locations (let, defn, ...). As you can see it easily eats 40% CPU of my machine (MacBook Pro (Retina, 15-inch, Mid 2015), 2,5 GHz Intel Core i7, 16 GB).

The input lag however only gets worse in certain situations it seems. Refactoring a variable name is definitely one case, #_ comments seem to have an effect to.

I'll look into yourkit, I had some evaluation licenses over the years. I might not get another.

[cursive-ide]

Ok, thanks - I've been meaning to investigate Flight Recorder, so this is a great excuse. I'll see if I can reproduce, but often with these sorts of problems it's hard without having actual work to do in the relevant files.

[thheller]

Excuse the stack image but something I noticed is that editing anything inside #js [foo] seems to cause huge lag. Even just deleting the thing via backspace.

Maybe there is a clue in there.

[cursive-ide]

@gnl BTW from which version did you upgrade to eap3? The previous EAP, or a stable version?

[gnl]

Sorry about the delay, been a busy few days. Here's the snapshot:

IU-171.4073.35_gnl_13.04.2017_18.35.39.zip

It's from eap6 and I'm doing some inline rename refactoring and then normally typing - the excessive input lag occurs during the refactoring. I wasn't able to reproduce the original behaviour that prompted me to open the issue, which was excessive input lag during normal typing.

I wasn't able to reproduce before upgrading Cursive either (with eap3) and I've been on the EAP channel for a while now.

I'll keep an eye on the memory, but so far I can't see anything excessive, it's moving between ~150 and ~220 out of 1484MB.

Update: Invalidating caches and restarting didn't help when the original issue was occurring.

[gnl]

After a couple hours of writing ClojureScript the memory indicator shows 1210 of 1484M, after having closed the REPL and all panes, if that's somehow relevant.

Updated: eap4

[cursive-ide]

@gnl If you click the memory indicator to provoke a GC, does that free it?

[gnl]

Didn't manage to get back to 1200MB, reached ~800M during the last session. On clicking the memory indicator it was released and went down to ~200M, but then quickly went back to 4-500 while just moving the cursor around the document.

[cursive-ide]

That's ok though - the churn is higher than I'd like, but if it drops back to 200M then a memory leak isn't the main issue.

[gnl]

Just now I experienced noticeably high input lag during normal typing and even higher when holding down backspace for example. I captured a new CPU snapshot:

IU-171.4073.35_gnl_16.04.2017_07.12.19.zip

[gnl]

And another one from excessive input lag during a small local inline rename refactoring of the fn-name-replacement binding in the following code:

        test-defn         (let [testfn-name         (symbol (str fun-name test-fn-suffix))
                                fn-name-replacement {fun-name testfn-name}]
                            (->> (list fn-deftype testfn-name docstring args prepost body)
                                 (remove nil?)
                                 (walk/prewalk-replace fn-name-replacement)))

IU-171.4073.35_gnl_16.04.2017_07.19.05.zip

[gnl]

Something just occurred to me - is it possible that under certain conditions something like inspection, highlighting etc., is triggered immediately on every keystroke rather than after a brief timeout and that there's been a change in this behaviour?

[cursive-ide]

@gnl Thanks for the dumps, I'll take a look at them tomorrow.

What you suggest is possible, but unlikely - that would have been a fairly major change at the IntelliJ level, not in Cursive. I suspect a more likely problem is Cursive holding on to a read lock during some operations for too long, so the Swing EDT gets blocked waiting for a write lock which affects UI responsiveness. I can't remember if the CPU dumps contain thread contention information, I'll check that tomorrow.

[cursive-ide]

@gnl So actually, your hunch was correct - at least, something similar. What I can't figure out is why this would suddenly have started happening, because the relevant code has not changed for ages. It's also caused by an interaction with IdeaVim, so possibly IdeaVim has changed in some way recently? I'm not sure.

Anyway, I've disabled the offending functionality in Cursive. I believe this should fix your issue, and from some testing it seems that perhaps it's no longer required any more. I'm not sure why - I'm checking with JetBrains. If this doesn't fix your issue, feel free to re-open this.

[gnl]

And to confirm - I've been on the EAP channel for as long as I can remember, only downgraded to 1.4.3 now in order to test and now I'm back on eap7.

[cursive-ide]

You're absolutely right, my apologies - I re-read your original but not your recent one after re-testing. Ok, I'll see if I can fix your issue and reproduce something like what @thheller is seeing. @thheller, if you could confirm when you've tried eap7 to see if the fix helps the problem you're seeing?

[thheller]

Just saw the bad input lag while editing a plain ~100 LOC .clj file on eap8. Just adding/removing a new line caused significant (>2sec) input lag before the next thing happened.

The issue is gone after restarting IDEA, can't really say how it got bad.

[thheller]

This might not be related to Cursive after all. I just saw the same input lag while editing a .java file. This was on the usual Clojure project with many open .clj files though so I can't really tell.

[cursive-ide]

I'm going to push this out. It seems unclear whether @thheller's problem is Cursive related, I've investigated a snapshot and I can't see anything suggesting Cursive is holding up the EDT there. I reproduced @gnl's problem, but that's been present for some time so I'm not going to hold up this release for it. I'm moving it into the next release and I'll look at it again there.

[thheller]

Yeah my issue is funky, don't hold back because of that. I misread the snapshot and the lazy seq is not on the UI thread.

I can't quite narrow down the issue, sometimes everything just works and then it doesn't. Maybe the issue gets more apparent when more people start using the latest code. Or it really is just me and my weird setup.

[pmoriarty]

I have a serious High CPU/UI lag issue since moving to 1.6.0. It happens frequently and I can't pinpoint a usage pattern that triggers it. Episodes seem to last minutes and CPU usage stays above 100% (of 800%) all that time. It often goes up to 400% for shorter periods. I'm also see spinning beach balls quite a bit.

Attaching CPU profiling dump. UI was very laggy for most of the time.

IU-171.4694.23_pmoriarty_04.07.2017_08.54.23.zip

IU-171.4694.23_pmoriarty_04.07.2017_09.26.49.zip

I'm on:
IntelliJ IDEA 2017.1.4
Build #IU-171.4694.23, built on June 6, 2017
...
JRE: 1.8.0_112-release-736-b21 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Mac OS X 10.12.5

[cursive-ide]

@pmoriarty So honestly, I'm not sure what's going on there. Your dumps show some CPU spikes:

But they don't look anywhere as extreme as what you're describing. And between those two or three spikes, there's basically tumbleweeds:

There is one thing that seems to be going on during some of those spikes, which is an interaction between the Groovy plugin and Cursive. I'm going to talk to JetBrains about that, since it looks like I'm doing some work which may be unnecessary there.

[cursive-ide]

I'm going to close this issue now. I've filed #1841 to track @gnl's issue which has been around for a while. I believe both @thheller's and @pmoriarty's are now fixed.

This issue ended up with several unrelated problems in it. If one of them reoccurs, please file a new issue and link this one rather than re-opening this one.