Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Multithreaded Job based Terrain Patch generation. #2163
There's a few things here, job, multithreading, memory usage reduction.
I've integrated a simple multithreading job library called JobSwarm and then I've parcelled up the terrain patch generation made it use as many hardware threads as you can throw at it :)
The library can be used for many other tasks. I've only implemented what I need for the terrain system.
Memory Usage Reduction:
Sat on Earth with the terrain detail to max I've reduced the footprint from ~700MiB to ~280MiB. Most of this was done by storing only the height instead of the full vector3d of positions, then reconstructing it in UpdateVBOs - a 2/3rds saving.
I think there's more that can be done here, the normal can be packed into 2 16bit values stored in a uint32 because it doesn't need all that precision, that could then be reconstructed either in UpdateVBOs. Or it could be turned into a texture and used in the shader to reconstruct the full normal. See this article on options. The heightmap could probably be done as a 32bit float too if I subtracted the clipCentroid length at generation time instead of leaving it until UpdateVBOs.
A lack of testing, I've done what I can, going in and out of the game, exiting and restarting etc. Going from planet to planet, system to system and it all seems robust... however.
On my quad core i7 with 8 hardware threads this thing is much faster at generating the terrain. On my dual core laptop, it might actaully be a bit slower :/
Summary? Um, it's great? please test! Kthxbye!
Okay, just pushed a fix for that issue.
Ahh, this didn't post - I lost power to part of my house and lost internet, but didn't realise. So this is slightly behind, but here we go...
I think switching now is fine. We're shouldn't rush this sort of thing, but with the release schedule changes there's certainly no reason to try and go too fast.
I'm not necessarily saying that we should use RobQueue (stupid name, screw you!). It was just an approach that didn't seem quite as messy as JobSwarm.
I moved the initialisation down because it theoretically should go under SDL initialisation if its going to use SDL threads (I say theoretically because SDL threads don't technically require it, but it reads nicer). That's one reason. The other was that I preferred a complete thread teardown at game exit because it didn't have a "cancel all threads" mechanism. But we've determined that we need that, so using the same queue and initialising it early should be fine.
Btw, I'm going to try and do the cancellation this week. I say "try" because there's release process retooling and savefile stuff to do this week as well.
Anyway I think I've found an optimisation to do with the finish lock.
SDL 1.2 has no control over thread priority. SDL 2 does. I vote for waiting until we switch to that (high on my list, though not much point until the UI is ported otherwise we'll be rewriting a whole bunch of stuff only to rip it out again. But I digress...)
Ok, lets merge it :)
added a commit
this pull request
May 6, 2013