New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excessive CPU usage #2

Open
johnniesong opened this Issue Apr 17, 2017 · 30 comments

Comments

Projects
None yet
5 participants
@johnniesong
Copy link

johnniesong commented Apr 17, 2017

wekan/wekan#718 this issue also happened on your docker ENV.

Downgrading from Node 6.10.0 to v4.8.1 solved the 100% CPU issue

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Apr 28, 2017

Fixed with downgrade to v4.8.1.

@xet7 xet7 closed this Apr 28, 2017

@mfshiu

This comment has been minimized.

Copy link

mfshiu commented Nov 19, 2017

My Wekan node is v4.8.4 , has the same issue, CPU will be 100% usage when over 20 clients.
Should I downgrade to v4.8.1 ?

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Nov 19, 2017

@mfshiu

You can try, please add comment did it help.

@xet7 xet7 reopened this Nov 19, 2017

@mfshiu

This comment has been minimized.

Copy link

mfshiu commented Nov 19, 2017

It's not worked.
Wekan make CPU loading very heavy when over 20 clients, always 100% CPU usage.

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Nov 19, 2017

@mfshiu

Ok, I think you are referring to this forum post:
https://discourse.wekan.io/t/cpu-utilization-problems-with-large-userbase/579/5

Wekan needs some profiling to identify bottlenecks etc.

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Nov 20, 2017

@GavinLilly

Did you do any profiling of Wekan with Kadira?

It could be possible that Meteor code tries to provide updates to all clients too fast, so some added delays in fetching and updating would made CPU load lower.

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Nov 20, 2017

@GavinLilly

If MongoDB is not a bottleneck as you said, then having search wekan/wekan#552 should be possible just with MongoDB.

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Nov 20, 2017

@mfshiu

I added to Roadmap Not started yet section "Try to fix Excessive CPU usage to encourage mfshiu to contribute his Wekan features in progress".

@mfshiu

This comment has been minimized.

Copy link

mfshiu commented Nov 22, 2017

Thank you for your encourage, I will try.

@GavinLilly

This comment has been minimized.

Copy link

GavinLilly commented Nov 23, 2017

I started setting up Kadira but didn't get round to profiling. Other things took priority. I will be able to look at it in mid-December. I agree with your hypothesis in that perhaps Meteor might be trying to do too much to update to each client.
For the time being I've doubled the number of running Wekan instances to 6 across 3 dual core VMs. It seems more stable but the instances are still reaching 100% load.

On the MongoDB side I've not observed any increased CPU or RAM load so some form of search could be possible. I'd be a little concerned about it searching using a cursor instead of an index though

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Nov 23, 2017

@GavinLilly

Yes mid-December is great, thanks a lot!

I did also add note to search items issue that search should be done by using index.

@mfshiu

This comment has been minimized.

Copy link

mfshiu commented Nov 24, 2017

I found the solution. The problem is "Frequent Subscriptions" in top of client\components\main\layouts.js

Meteor.subscribe('boards');
Meteor.subscribe('setting');
Meteor.subscribe('user-admin');

I move these code to top of client\components\boards\boardList.js

BlazeComponent.extendComponent({
onCreated() {
console.log("{boardList.js} (onCreated) begin");
Meteor.subscribe('boards');
Meteor.subscribe('setting');
Meteor.subscribe('user-admin');
},

CPU is no longer too high, from 100% down to 30% - 50%.
The subscriptions at top of layouts.js will make boards data sync every where, it's the reason that cause CPU heavy when user number is increased. To move it to creation time of boardList.js, there is the only place that boards data needed, all boards data will not sync when user enter a board or card.
I will try to update the code or you can told me how to do that.

@mfshiu

This comment has been minimized.

Copy link

mfshiu commented Nov 24, 2017

It seems I have no right to push the updates.
remote: Permission to wekan/wekan.git denied to mfshiu.' 'fatal: unable to access 'https://github.com/wekan/wekan.git/': The requested URL returned error: 403

How to apply or you can help me?

@mfshiu

This comment has been minimized.

Copy link

mfshiu commented Dec 10, 2017

Use this package in Wekan is VERY important:

mixmax:smart-disconnect

Meteor connection cost is very expensive, if your MongoDB contains large data, your active users are over several hundreds, you must use the package to auto disconnect user when they leave their focus on Wekan over 1 minute. I read the article by Eliezer Steinbock, then make my Wekan CPU ultra down.

@GavinLilly

This comment has been minimized.

Copy link

GavinLilly commented Dec 11, 2017

Is it possible to integrate smart-disconnect into the published release of Wekan after downloading or would I need to pull the source code and integrate before build/deploy?

@xet7 xet7 reopened this Dec 11, 2017

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Dec 11, 2017

@GavinLilly

Yes, I will try to include smart-disconnect with Wekan.

xet7 added a commit to wekan/wekan that referenced this issue Dec 12, 2017

Add mixmax:smart-disconnect to lower CPU usage when
browser tab is not selected. Thanks mfshiu !
Related wekan/wekan-mongodb#2

xet7 added a commit to wekan/wekan that referenced this issue Dec 12, 2017

Add mixmax:smart-disconnect to lower CPU usage when
browser tab is not selected. Thanks mfshiu !
Related wekan/wekan-mongodb#2

xet7 added a commit to wekan/wekan that referenced this issue Dec 12, 2017

Add mixmax:smart-disconnect to lower CPU usage when
browser tab is not selected. Thanks mfshiu !
Related wekan/wekan-mongodb#2
@GavinLilly

This comment has been minimized.

Copy link

GavinLilly commented Dec 15, 2017

I'm not convinced that smart-disconnect was correctly bundled with the v0.62 release:

$ cat programs/server/packages/mixmax_smart-disconnect.js
(function () {



/* Exports */
if (typeof Package === 'undefined') Package = {};
Package['mixmax:smart-disconnect'] = {};

})();

As you can see no logic is included in the file. Is there another way to check if a package was included correctly?

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Dec 15, 2017

@GavinLilly

I only added package:
https://github.com/wekan/wekan/blob/devel/.meteor/versions#L95
https://github.com/wekan/wekan/blob/devel/.meteor/packages#L83

I don't know what else @mfshiu did to get it working.

At package page there is only info about IronRouter, not FlowRouter that Wekan uses:
https://atmospherejs.com/mixmax/smart-disconnect

If somebody knows how to get this working, please add pull request.

@GavinLilly

This comment has been minimized.

Copy link

GavinLilly commented Jan 2, 2018

Actually even though there doesn't seem to be any logic in that JS file I have observed that when I switch to a Wekan tab in Chrome I briefly see the message near the top of the page saying "Wekan is loading..." so it seems the smart disconnect package is doing it's job after all.

In addition, performance in our Wekan platform appears to be improved recently though that could also be affected by decreased traffic over the Christmas break. I'll report back if we experience further issues as traffic ramps up

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Jan 2, 2018

@GavinLilly

Yes I noticed that behaviour also with Firefox previously, so smart disconnect seems to work.

@xet7

This comment has been minimized.

@xet7 xet7 referenced this issue Feb 4, 2018

Closed

LDAP Integration #119

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Feb 4, 2018

Links for some future optimizations research:

@GavinLilly

This comment has been minimized.

Copy link

GavinLilly commented Apr 3, 2018

We've fixed our CPU usage problem today with an environment change around Wekan. I wasn't aware during implementation that if you're using more than 1 instance of Wekan (or any MeteorJS based tool) you're supposed to set MONGO_OPLOG_URL as an environment variable. Without setting it, Meteor will perform a pull-and-diff update of it's dataset. With it, Meteor will update from the OPLOG. See here

After setting MONGO_OPLOG_URL=mongodb://<username>:<password>@<mongoDbURL>/local?authSource=admin&replicaSet=rsWekan the CPU usage for all Wekan instances dropped to an average of less than 10% with only occasional spikes to high usage (I guess when someone is doing a lot of work)

So as you can see it's not a problem with Wekan but how it's implemented in a given infrastructure. Apologies for the wild goose chase in previous months. Perhaps there should be an entry on the Wiki so users know to set this if they're using more than 1 instance of Wekan. Also perhaps it could be added to the docker compose yaml?

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Apr 3, 2018

@GavinLilly

What Wekan, Node and MongoDB version you use?

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Apr 3, 2018

@GavinLilly

What kind of infra you use? Number of instances, Docker, etc?

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Apr 3, 2018

Nevermind, thanks for info, I'll add it :)

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Apr 11, 2018

@kentonv

Currently Wekan get's complaints about 100% CPU usage when there is thousands of users on powerful servers, with this kind of setup at AWS:
https://github.com/wekan/wekan/wiki/AWS

Wekan uses Meteor 1.6.0.1, but most likely same problem is at 1.6.1.

There is currently these GitHub issues going on about this here:
meteor/meteor#9796

Referencing to this issue you had:
meteor/meteor#7747

Do you have ideas how this could be fixed in Wekan or Meteor ?

@kentonv

This comment has been minimized.

Copy link

kentonv commented Apr 16, 2018

@xet7 I commented on the other thread. You can try my patched node (linked there) and see if it helps. (If you don't want to compile yourself, copy /opt/sandstorm/latest/bin/node from your Sandstorm install.)

xet7 added a commit to wekan/wekan that referenced this issue Apr 16, 2018

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Aug 28, 2018

Please test newest Wekan v1.39. There has been speedups in Wekan v1.37 and v1.38. CPU usage also seems to be lower.
https://github.com/wekan/wekan/blob/devel/CHANGELOG.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment