Improving performance for extremely large datasets

[diragb]

hey, i'm attempting to display a large dataset (117,927 nodes and 117,921 links), but WebGL runs out of memory after the first few seconds. is there any way to optimize the model or the animation to reduce/eliminate crashes, fps throttling, memory hogging?

[diragb]

used the 3rd advice from #181 of setting warmupTicks={100} and cooldownTicks={0}, also turned off bloom postprocessing. it's working now, but the fps is still choppy + it took around 5 minutes before i could see the structures (it was a white screen before that).

[diragb]

a little good news, the structure finally renders with the post-processing
it's still very laggy though (due to obvious reasons), and takes a long time to load (5-7 mins)

here are some snaps!

[vasturiano]

Those screenshots look stunning!

You're probably pushing your hardware to its limits by rendering so many objects. 😄
I think the main things that you could do to alleviate the pressure is to simplify the objects themselves. A few examples I can think of:

• Not using curved lines. Straight lines are simpler objects.
• Reducing the resolution of the spheres, using .nodeResolution.
• Also the less post processing effects you have, the easier it is but I see you already tried deactivating the bloom pass filter.

[diragb]

hey, i used your suggestions and there was a moderate performance boost. i'll be using a few of my friends' powerful computers to render the model and hopefully take a few videos of it. i'm closing this issue now, i'll be commenting a link of our video (i'll credit you) as soon as it's done. thanks for the help!

[kongweiying2]

hey, i used your suggestions and there was a moderate performance boost. i'll be using a few of my friends' powerful computers to render the model and hopefully take a few videos of it. i'm closing this issue now, i'll be commenting a link of our video (i'll credit you) as soon as it's done. thanks for the help!

This looks pretty cool. Would you be able to share the source for the bloom post processing by any chance? I'm having trouble getting it to work.

[vasturiano]

@TheAussieStew there is an example of it here.

Relevant code:

react-force-graph/example/bloom-effect/index.html

Lines 28 to 34 in 153ab1f

	useEffect(() => {
	const bloomPass = new UnrealBloomPass();
	bloomPass.strength = 3;
	bloomPass.radius = 1;
	bloomPass.threshold = 0.1;
	fgRef.current.postProcessingComposer().addPass(bloomPass);
	}, []);

[kongweiying2]

Ah thank you! I looked at that example but I'm having trouble getting useRef to work with Typescript. Looks like I'll just have to experiment.

[diragb]

a meaningful update.

me and my friends used this library and made this video some time back.

this 3d graph only had around 10k to 20k nodes. some of the issues, as i mentioned here, were FPS drops. to fix this issue, we used an open source AI project that transforms low FPS footage to high FPS (DAIN) -- however this proved to be problematic since there were lots of artifacts.

eventually, we settled on producing this video with the shoddy footage and calling it a day. it was only a month later that we thought about timelapse-ing the whole project. so, essentially, i wrote a small script with rotated + panned + zoomed on the graph extremely slowly, slapped on OBS recorder on top of it, recorded for an hour and sped up the footage by 10x to 100x. the result was nothing short of fascinating.

we still haven't released this footage yet, because we got distracted and might work on it in the future, but to anyone else wanting to make a video on extremely large 3d graphs using this library -- this is the best way you can do it, and the results will be insanely beautiful.

i'll add a short video clip soon, once i get hold of my old hard drive. :)