New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wekan hung up after upload big json #1488

Closed
swood opened this Issue Feb 16, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@swood
Copy link

swood commented Feb 16, 2018

Unfortunately, I can’t send in the public space this json because it contains private data. But I can tell you count of keys: approximately 28000 pairs.
And the size of json: about 3.6Mb.
And also I have profiler output of nodejs after I tried last time to import this json. I think the very interesting strings there:

[JavaScript]:
ticks total nonlib name
1341 0.6% 5.2% LoadIC: A load IC from the snapshot
759 0.3% 3.0% LazyCompile: * /build/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:390:18

[Bottom up (heavy) profile]:
Note: percentage shows a share of a particular caller in the total
amount of its parent calls.
Callers occupying less than 1.0% are not shown.

ticks parent name
187232 82.6% /opt/nodejs/bin/node
29983 16.0% LazyCompile: * packages/meteor.js:844:25
29981 100.0% LazyCompile: *processImmediate timers.js:697:26
18203 9.7% LazyCompile: *Query.toBin /build/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/commands.js:89:33
18112 99.5% LazyCompile: *Pool.write /build/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:928:32
18110 100.0% LazyCompile: *nextFunction /build/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/cursor.js:518:28
18109 100.0% LazyCompile: *Cursor.next /build/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cursor.js:257:33
18001 99.4% LazyCompile: *fn /build/programs/server/node_modules/fibers/future.js:82:20
17319 9.3% /opt/nodejs/bin/node
5404 31.2% LazyCompile: *stop /build/programs/server/packages/ddp-server.js:2029:22
2726 50.4% LazyCompile: * /build/programs/server/packages/mongo.js:1283:34
2726 100.0% Builtin: ArrayForEach
2726 100.0% LazyCompile: *stop /build/programs/server/packages/mongo.js:1282:20
2678 49.6% Function: ~ /build/programs/server/packages/mongo.js:1283:34
2678 100.0% Builtin: ArrayForEach
1877 70.1% LazyCompile: *.each..forEach packages/underscore.js:136:43
801 29.9% LazyCompile: *stop /build/programs/server/packages/mongo.js:1282:20
2902 16.8% LazyCompile: *_.isEmpty packages/underscore.js:1033:23
2900 99.9% Function: ~stop /build/programs/server/packages/ddp-server.js:2029:22
2900 100.0% Function: ~ /build/programs/server/packages/mongo.js:1283:34
2900 100.0% Builtin: ArrayForEach
1453 8.4% LazyCompile: *Buffer.write buffer.js:855:34
1453 100.0% LazyCompile: *Query.toBin /build/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/commands.js:89:33
1453 100.0% LazyCompile: *Pool.write /build/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:928:32
1453 100.0% LazyCompile: *nextFunction /build/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/cursor.js:518:28
1059 6.1% LazyCompile: *_pollMongo /build/programs/server/packages/mongo.js:2150:24
1059 100.0% Function: ~ /build/programs/server/packages/mongo.js:2116:40
1059 100.0% LazyCompile: * packages/meteor.js:845:20

It’s look like the frontend tries to update of each card in realtime and generating a lot of requests to backend. Does this version make sense?

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Feb 16, 2018

@swood

Yes. Meteor.js framework parses MongoDB changes realtime and updates everything realtime.

For that amount of data, you need to have Wekan server installed similarly like production cluster at AWS with high-speed connectivity for Wekan to be useable. I would also recommend adding redis oplog as described in the link above.

When importing that large board, it's very likely that Wekan tries to get a lot of attachments from Trello, and if server is not powerful enough, you could have timeouts, or have some data missing.

@swood

This comment has been minimized.

Copy link
Author

swood commented Feb 18, 2018

Could you please tell how to enable redis-oplog for wekan, because these article doesn't content exhaustive about it. And also I've found information about mixmax:smart-disconnect. Does it help?

@xet7

This comment has been minimized.

Copy link
Member

xet7 commented Feb 19, 2018

@swood

I have not tried redis-oplog yet, because without that performance has been enough for thousands of users.

Wekan already has mixmax:smart-disconnect added. You see it that Wekan only updates itself on webbrowser when browser tab is active. When you use other browser tab, Wekan will timeout and be idle, and when you switch back to Wekan tab, Wekan will connect to backend again and update webpage.

For your use case, having Wekan installed on one powerful enough server installed at AWS or elsewhere (other than your laptop) should be enough to get that big Trello JSON imported. You can add comment to this issue about did it work for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment