80-100% CPU usage all the time

[valueof]

My copy of LightTable seems to be constantly using 80-100% of my CPU and I have no idea why.

$ top -o cpu
Processes: 212 total, 2 running, 5 stuck, 205 sleeping, 1180 threads                                      15:44:59
Load Avg: 3.18, 8.62, 8.46  CPU usage: 15.72% user, 9.11% sys, 75.16% idle
SharedLibs: 18M resident, 11M data, 0B linkedit.
MemRegions: 67626 total, 3046M resident, 71M private, 1425M shared. PhysMem: 11G used (1994M wired), 4339M unused.
VM: 547G vsize, 1066M framework vsize, 3169146(0) swapins, 3504586(0) swapouts.
Networks: packets: 16954010/9810M in, 8503712/1823M out. Disks: 2046192/46G read, 3012196/111G written.

PID    COMMAND      %CPU  TIME     #TH   #WQ   #PORT #MREGS MEM    RPRVT  PURG   CMPRS  VPRVT  VSIZE  PGRP  PPID
48377- node-webkit  76.8  84:48.93 16    0     149   1363   458M-  457M-  0B     0B     1090M  1874M  48375 48375
...

I'm on OS X 10.9.1, reproduced both on my Air at home and Pro at work. I'll be happy to debug further—what information is helpful in this case?

[ibdknox]

Does this happen if the workspace is empty?

[valueof]

Of course, as soon as I filed a bug, it stopped happening. I did some further investigation:

• MBA: I have a workspace with a tiny Clojure project in it. It doesn't seem that having a workspace makes any difference. However, CPU spiked once I added a connection.
• MBP: I use it for Firefox dev, only one workspace with a sub-tree that doesn't have any .git or build directories in it. CPU spikes when I check out the code (updating the cache?) and building Firefox. The latter is weird since nothing in my workspace'd directory changes during the build: no writes, only reads.

[geoffrys]

MBP OSX here.

After a long day of using the same LT repl connection I noticed spikes to 100% CPU every time I evaluated a form. This spike stayed for about 20 seconds after the form had produced a value.

I removed all files from the workspace, exited all open files, and created (and saved) a new empty file, in this file, using the same connection, I evaluated (+) and noted the same results (long CPU spike).

[ibdknox]

@geoffrys I think this might've been related to the way we were doing auto-complete for clojure which eventually gets pretty unwieldy. I just pushed a new version of the Clojure plugin, can you update and see if that makes things better?

[valueof]

I got the spike again (node-webkit taking ~80% CPU). This time, however, I opened web inspector and recorded some cpu profiler data. Unfortunately, the profile shows 99.9% idle time.

node-webkit bug, perhaps?

[valueof]

Huh, I just quit LT but node-webkit process was there for a good minute or so.

[mynomoto]

Coincidentally I was reading this and noticed it happening.

I'm on 0.6.2 in arch 64bits. I tried the package in aur for a while and noticed that this was happening all the time. Switched back to manually install and thought that it was gone until now. Restarting helps but it's annoying. All plugins are updated.

[michaelphines]

I've also been experiencing this bug in my Clojure project. It'll peak at 99%, but even when it's mostly behaving the CPU hovers around 80%. My project is very small (<25 files).

Instarepl is off, but I do have an nrepl connection. CPU load appears to go down when I start working in other apps, but then ramps back up as I type in the editor and continues for a bit.

The most common symptom is that typing is slow to respond, and sometimes keys will repeat. Running individual commands with Cmd-enter seems to behave, though.

I'm on a Macbook Air 4,2 running Mavericks 10.9.1

About page says:
Light Table version 0.6.4
Binary version 0.8.4

I installed from source, and I'm running from the deploy folder.

The nrepl commands from the connections panel looks like this:

lein-light-nrepl
:editor.eval.clj
:docs.cljs.search
:editor.cljs.hints
:cljs.compile
:editor.eval.clj.cancel
:editor.cljs.doc
:editor.clj.hints
:editor.eval.cljs
:editor.eval.clj.sonar
:docs.clj.search
:editor.clj.doc

Here's a list of my plugins. I had more, but I tried to remove a bunch of them in hopes that one of them was the problem.

Bracket glow 0.0.1
CSS 0.0.3
Clojure 0.0.8
GBlame 0.0.6
HTML 0.0.2
HTML Live 0.0.1
JSHint 0.0.2
Javascript 0.0.1
Palette 0.0.1
Paredit 0.0.4
Paredit-Plus 0.0.3
Python 0.0.3
Rainbow 0.0.8
Recall 0.0.6
* Ruby 0.0.1 
Subpar 0.0.1
UUID 0.1.0
** Vim 0.0.4
Whitespace 0.0.1

*  I have this turned on
** Which I installed once, but now can't remove via the 
   plugins manager because update doesn't work and
   the update function hides the remove function.

I'm willing to do some debugging, but I've never worked with node-webkit.

[ibdknox]

subpar destroys typing performance

[michaelphines]

Good to know. I'll remove that, restart, and give it a shot. Are paredit or paredit-plus similarly a problem?

I'll report back, since I was pretty sure removing subpar was one of the things I tried. I really don't need it though.

[ibdknox]

paredit and paredit-plus are both fast

[michaelphines]

Alright. After about an hour without subpar, I'm still consistently seeing the problem. Activity Monitor reports node-webkit Helper is currently running at 92.2% CPU.

[michaelphines]

Nevermind my previous comment... this worked to disable autocomplete, but still is doing nothing for the CPU problem:

 {
  :- {:app [:lt.plugins.auto-complete/init]
     :editor [:lt.plugins.auto-complete/intra-buffer-string-hints
              :lt.plugins.auto-complete/textual-hints
              :lt.plugins.auto-complete/async-hint-tokens
              :lt.plugins.auto-complete/show-hint
              :lt.plugins.auto-complete/remove-on-move-line
              :lt.plugins.auto-complete/remove-on-scroll-inactive
              :lt.plugins.auto-complete/auto-show-on-input]
     :editor.behaviors [:lt.objs.settings/on-behaviors-editor-save
                        :lt.objs.settings/eval-settings
                        :lt.objs.langs.behaviors/behavior-hints
                        :lt.objs.langs.behaviors/show-info-on-move
                        :lt.objs.langs.behaviors/on-changed
                        :lt.objs.langs.behaviors/parsed
                        :lt.objs.langs.behaviors/behavior-hint-pattern
                        :lt.plugins.auto-complete/auto-show-on-input]
     :editor.keymap [:lt.plugins.auto-complete/auto-show-on-input]
     :hinter [:lt.plugins.auto-complete/select :lt.plugins.auto-complete/escape!
              :lt.plugins.auto-complete/select-unknown
              :lt.plugins.auto-complete/line-change]}}

[ibdknox]

Does the CPU only go up while typing?

[michaelphines]

• If I leave it alone for about 5 minutes, the CPU will settle down to 5-10%
• I have VIM enabled, and if I start just using simple motion keys (h j k l), it'll spike to 30%. I moved around the file for about a minute and it looked like it wasn't going much higher than that.
• When I change to insert mode and start typing, it jumps to 80%-90% almost immediately. For example, this time it was just after typing "(hello world my name is m)"
• If I leave the app it'll still take about 5 minutes or more to completely settle down.

Also, this was all after a restart of the app so I didn't have an nrepl connection running.

[ibdknox]

Hm. Well we can run a js profile to see what's going on, as it sounds like something that'd be caused by one of the plugins. In the command pane type dev-inspector and press enter. That will open the chromium dev tools, click profiles, and then start a javascript profile. If you take a screenshot of a somewhat zoomed in flamechart view of the profile, we'll probably be able to tell pretty quickly.

[michaelphines]

Cool. I saw people talking about the dev-inspector but I couldn't figure out how to open it. I didn't realize it's hidden.

This is weird: While typing, it's pretty clear that things are happening. Here is a keystroke from a profile while I was typing. Most of the flame chart looks like this. One keystroke here took 200ms to process if I'm reading this right. Not great, but I'm not identifying a clear cause from the flame chart.

However if I profile the javascript while I'm not typing I get barely anything. So little that the flame chart is almost useless. The CPU usage is still around 80% (adding the profiler overhead bumps it over 100%). The whole thing seems to be taken up by (program) which I'm assuming is Chromium?

[michaelphines]

Interesting. I just noticed these showed up in the console when I start dev-inspector. Not sure if it's related. It looks like it's just failing to load sourcemaps.

Could not load content for file:///Users/mhines/Projects/LightTable/deploy/core/clojure_compiled.js.map : Loading resource for inspector failed
inspector.js [7455] contentLoaded
Could not load content for file:///Users/mhines/Projects/LightTable/deploy/core/vim_compiled.js.map : Loading resource for inspector failed
inspector.js [7455] contentLoaded
Could not load content for file:///Users/mhines/Projects/LightTable/deploy/core/uuid_compiled.js.map : Loading resource for inspector failed
inspector.js [7455] contentLoaded
Could not load content for file:///Users/mhines/Projects/LightTable/deploy/core/palette_compiled.js.map : Loading resource for inspector failed
inspector.js [7455] contentLoaded
Could not load content for file:///Users/mhines/Projects/LightTable/deploy/core/gblame_compiled.js.map : Loading resource for inspector failed
inspector.js [7455] contentLoaded
Could not load content for file:///Users/mhines/Projects/LightTable/deploy/core/paredit-plus_compiled.js.map : Loading resource for inspector failed
inspector.js [7455] contentLoaded
Could not load content for file:///Users/mhines/Projects/LightTable/deploy/core/css_compiled.js.map : Loading resource for inspector failed
inspector.js [7455] contentLoaded

[ibdknox]

yeah, those flamecharts are unfortunately not very useful. Let's rule out (or verify) plugins for sure.

Can you try renaming your ~/Library/Application Support/LightTable/plugins/ to plugins2 (or whatever) and creating a fresh plugins dir. Then restart, that should give us a stock version of LT.

[ibdknox]

Do you have any symlinks in your project?

[michaelphines]

No symlinks. I have some raw json file exports that are 20K each all in one line, but they're in a folder that's .gitignored and it looks like LightTable is respecting that in the workspace.

When I rename my plugins folder, it seems to resolve the problem where node-webkit stays at high CPU while I'm not using it. Typing still spikes up to 90%, but it responds relatively snappily, and returns to 10% within a few seconds.

The interesting thing I noticed is that there are two node-webkit helpers that are started:

/Users/mhines/Projects/LightTable/deploy/LightTable.app/Contents/Frameworks/node-webkit Helper.app/Contents/MacOS/node-webkit Helper --type=renderer --enable-experimental-web-platform-features --disable-threaded-compositing --js-flags=--harmony --no-sandbox --lang=en-US --nodejs --working-directory=/Users/mhines/Projects/LightTable/deploy --child-clean-exit --disable-accelerated-video-decode --channel=92728.0.1797370189
/Users/mhines/Projects/LightTable/deploy/LightTable.app/Contents/Frameworks/node-webkit Helper.app/Contents/MacOS/node-webkit Helper --type=gpu-process --channel=92728.1.417963958 --no-sandbox --supports-dual-gpus=false --gpu-driver-bug-workarounds=0,10,13,18 --gpu-vendor-id=0x8086 --gpu-device-id=0x0116 --gpu-driver-vendor --gpu-driver-version

The one that ends up pegging the CPU is the process started as node-webkit Helper --type=gpu-process

I guess I can start adding back plugins one by one and report back if I notice one in particular is the problem, but in that case maybe the issue belongs elsewhere.

[afhammad]

+1

[michaelphines]

Interesting, I've been using LightTable for the past couple days with only the Vim and Paredit-Plus plugins installed, and the app is slowly starting to show the same symptoms. This is how I got the problem initially.

It starts out as a little bit of an annoyance: typing is slow sometimes.
Then over a couple hours typing becomes slower and slower.
In a day or so it's more of an annoyance and the node-webkit helper process starts taking CPU while in the background for a couple seconds after switching the app, and then progresses from there. When I first posted about this it was after about a week of using LightTable, and at that point it had become unusable.

I'll try to use it for a while with no plugins, but I might have to switch back to something else since vim keybindings are big for me.

[ibdknox]

I can pretty much guarantee it's not the vim plugin. Can you take a heap snapshot when it starts to slow down and tell me how big the heap is? Long running degradations points to something memory related.

[ibdknox]

It most likely has to do with leaking file watchers, I bet.

[michaelphines]

The problem is most consistent when I restore the old plugins directory from when the problem was worst. This is what it looks like:

It appears the heap is 21MB. Other than that, I don't see anything incredibly interesting, but maybe I'm not looking in the right place. I'll take a heap snapshot again next time it's acting up with the current plugins directory.

[michaelphines]

Here it is today with vim as the only additional plugin. Similar heap size. This time I switched to containment view, in the hope that it's slightly more informative:

The difference between heap sizes when the app is started to when the problem becomes noticeable is ~22MB - ~18MB = 4MB, at least in the few times I've checked.

Sorry I'm not posting the export of the full heap, but I'm worried it might contain proprietary code. When I close the workspace I'm working on and open another workspace (the LightTable repo for instance) the problem goes away, at least for the short term. Then, when I open my project back up again, it's not displaying the problem anymore. At least for a while.

This gives me a theory. Maybe, the reason that the problem is so predictably reproducible when I restore my old plugins directory is that one of those plugins is the "Recall" plugin. That plugin attempts to restore the workspace (including open files) between restarts. I haven't dug into how it does that, but it may be a clue. The problem seems to build up when I've left LightTable open for a while, so now since I'm not "recalling" my previous workspaces, restarts seem to solve the problem. When I restore my previous plugins directory with the Recall plugin, it's restoring the "workspace" somehow, and that's why I see the problem immediately with all plugins instead of having to wait for the problem to occur.

[michaelphines]

I think I found one memory leak. It's in the BEncoding module. I guess the nrepl uses BEncoding as the transport by default? Anyway, the buffer is very large in my problem heap dump vs. the heap dump without a problem:

Problem dump:

No problem:

[michaelphines]

Instead of sending more screenshots, I can just summarize other things I'm noticing. These classes appear to be defined in CodeMirror. They're showing abnormally high object counts.

"Bad" is from the heap dump when I'm seeing the problem.
"Control" is one I took when I wasn't seeing the problem.

Constructor	Bad Object Count	Control Object Count	Bad Shallow Size	Control Shallow Size	Bad Retained Size	Control Retained Size
TextMarker	324	1	15,516	12	22,240	532
Pos	620	6	12,392	112	26,808	136

[mynomoto]

I'm not seeing this for a while now. (I'm on master and every plugin in the latest).

@michaelphines Is this still happening to you?

[pnutus]

I get close to 100% cpu usage with LT 0.6.7 in the background even (no computation, no typing). This is while using the Julia plugin (0.9.1) on OS X 10.9.5.

[hamoid]

Happening to me in Ubuntu on a JavaScript, HTML, CSS project. But I'm not opening any browsers inside LT, but using Chromium and Firefox side by side.

From the 7 processes LT has running, I see that the one using cpu is the one that ends with "ternserver.js --harmony". Does that say something useful?

[joshuafcole]

Hey hamoid, thanks for the more detailed report. Certain third party olugins do a lot of heavy lifting, such as the ternjs plugin. If youre having iasues with that being very intensive, you might try removing it to ensure itd the problem, and if it is cross-post the issue against its repo.

[hamoid]

Ok, I'll report on the ternjs plugin. It's not that it's just heavy lifting. It's stuck. It shows 25% because that's one of the four cores. It means 100% on one core. Even if I close LT it stays there, and if start it again, I may end with several instances consuming one core each.

[icorderi]

Any updates on this issue? My LT is freezing every few seconds, typing, saving, opening, pretty much everything pegs the cpu. I can't narrow it down to what is causing this.
Does anyone have a workaround or tips to alleviate this?

Running OSX 10.10.1

[ibdknox]

We're working on moving to the atom-shell to help alleviate the Yosemite issues people are seeing. This thread on the mailing list talks about how to try it out, though there are a few rough edges on it right now.

[jdf-id-au]

Is the nw problem fixed now? (Does LT use CSS3 animations?)

[KTKate]

I downloaded & installed LT for the first time 2 days ago. Opened a file in it and got distracted. Never did anything else in LT. Today I am working in another app and suddenly my laptop fans start going like crazy. Turns out to be this node-webkit helper. I close light table and it takes about 3 minutes for the process to go away.

I don't think I had any plugins unless there are any that run by default.

OSX 10.10.3

[kenny-evitt]

@jdf-id-au @JKGisMe If you could, try building and running the atom-shell branch version.

[BenjaminVanRyseghem]

I used to experienced something similar quite often.

Now that I am using the atom-shell branch, I will report it here if I bump into this again 😄

[kenny-evitt]

atom-shell is merged into master; just run a build of the latest source.

[Ghassan]

I am running LT 0.7.2 on OSX 10.10.4 MBP 16 GB RAM and I get:

1. the user_compiled.js error which I don't know what it is.
2. node-webkit Helper is choking the entire Mac. usually runs about 140% CPU time.

I have to frequently restart sometimes my entire Mac.

[GetContented]

This project hasn't had a commit for 26 days, sadly.

[kenny-evitt]

@Ghassan Would you create a new issue for [1] and provide enough details for someone to reproduce the error you're observing?

But before you do that, please try building Light Table from the latest source of the master branch. The latest source version shouldn't have this issue as LT was migrated from node-webkit to Atom/Electron (precisely to fix this and other similar issues).

[kenny-evitt]

@GetContented Pull requests welcome! And keep in mind that this project is spread out over multiple repos; most of the interesting functionality is implemented in separate plugins, and there has even been an update to one of them since the last commit for this repo.

There are a few of us working on packaging up the latest version as a new release, but we're not the original contributors so it's slow going for us to get up (and stay up) to speed on everything.

[GetContented]

@kenny-evitt understood! It's hard to know this stuff as an "ordinary observer"... no blog updates for over 6 months, too. Would be good if there were slightly more frequent updates there. Not trying to be overly critical, just letting you know what it seems like. Anyway, this probably isn't the right spot to make this comment (not sure where is, though).

[kenny-evitt]

@GetContented There's a few of us somewhat active on the Gitter room for this, the main GitHub repo. The Google Groups group is also suitable for general questions and discussion.

[Ghassan]

@kenny-evitt I downloaded the main branch, looked around everywhere possible for instructions on how to build LightTable from the source but couldn't find any. Went ahead and ran build.sh under script (don't know if this is all there is though?). then I executed "light" under builds which I noticed it was created after I ran the script.

The editor opened had "About Atom" as the first menu item under LighTable menu, but does not respond to "^-space" to open the command menu and "View" menu doesn't have light table options.

So I wasn't able to use the editor. I did notice the switch to Atom/Electron.

[kenny-evitt]

@Ghassan Running script/build-app.sh is all you need to do to build – assuming you have Node.js and npm installed.

Instead of running script/light.sh, I usually run the build output executable directly; in your case the LightTable.app file under builds\lighttable-0.8.0-mac.

[Ghassan]

I have node 0.12.7 and npm 2.12.1. Running build-app.sh doesn't do much. Just the same behavior. I tried to delete "builds" directory but it did not recompile. Any advice? also pointers to docs on how to build from source with optional commands? Thanks a lot.

[kenny-evitt]

@Ghassan I was wrong in writing earlier that you only need to run script/build-app.sh – you need to run script/build.sh at least once. build-app.sh just packages the build files.

If you can, try building the latest source version again.

also pointers to docs on how to build from source with optional commands?

I'm not sure what you mean by "optional commands".

[Ghassan]

Ah, ok, thank you so much for letting me know. I'll try that.

-Ghassan
[image: www.linkedin.com/in/ghassanayesh]
http://www.linkedin.com/in/ghassanayesh
http://ghassan-ayesh.blogspot.com/

On Fri, Oct 2, 2015 at 11:34 AM, kenny-evitt notifications@github.com
wrote:

@Ghassan https://github.com/Ghassan I was wrong in writing earlier that
you only need to run script/build-app.sh – you need to run script/build.sh
at least once. build-app.sh just packages the build files.

If you can, try building the latest source version again.

also pointers to docs on how to build from source with optional commands?

I'm not sure what you mean by "optional commands".

—
Reply to this email directly or view it on GitHub
#1088 (comment)
.

[kenny-evitt]

@valueof @geoffrys @mynomoto @michaelphines @afhammad @otmezger @GetContented @davidssmith @mrobbetts @AlBaker @marcusps @pnutus @hamoid @icorderi @jdf-id-au @JKGisMe @Ghassan If you would, please confirm whether you're still having this problem with the 0.8.0-alpha version. That version includes the migration from node-webkit to Electron (Atom Shell).

[cldwalker]

We've had 0.8.0 out for a month and haven't heard back. Closing until we have reproducible steps that can lead to a fix

[GetContented]

Not sure about others, or even if you want to know this, but I gave up using light table ages ago.

[davidssmith]

@GetContented Me too.