Far map and improved map transfer and rendering (WIP)

[celeron55]

This does two things like the title says: far map rendering and improved map transfer, and includes a couple of supporting commits.

I request testing and comments about whether this is upstream-worthy. This is causing a lot of conflicts at the moment because people are eager to work on the map transfer which this reworks heavily.

You can see the original spammy commit log by looking at celeron55:far_map_wip_2.

This is my to-do list at the moment of posting this PR. All of the items on this list are strongly "maybe", subject to changes and/or practically undecipherable.

- Check how empty MapBlocks and FarBlocks are handled and possibly improve it

? Make it so that empty FarBlocks don't consume huge amounts of memory

- Rename enable_far_map to enable_far_map_system to indicate that it completely
  switches it into and out of use

- Merge gregorycu's patch

- Rename TOSERVER_SET_WANTED_MAP_SEND_QUEUE to TOSERVER_SET_MAP_SENDS

? Add max_simultaneous_block_sends to TOSERVER_SET_WANTED_MAP_SEND_QUEUE

- Unload FarBlocks sometimes
    - May have to use IVideoDriver::removeHardwareBuffer

? Increase default target FPS to 60

? Should FarBlocks be aligned with map generator chunks?

? Make FarBlocks to be scene nodes so that irrlicht will render them on its own

- Maybe store zlib-compressed data on ServerFarBlocks

- The server has to do occlusion culling for FarBlocks due to network usage
    * Or maybe not?

- The server must be able to tell the client that a FarBlock was culled and may
  not be actually empty

- Add a setting for disabling far map on server side

- Use frustum culling when rendering FarBlocks

- The client has to do occlusion culling for FarBlocks due to the GPU having a
  limited amount of space for VBO meshes

- Inform FarMap about blocks that were found to not exist too

- far_map_enable_maximum_resolution: Basically the "ultra" setting of Minetest
    - ServerFarMapPiece::content_max_res

- Add a minimal logging subsystem for map transfer that can be turned on

- Profile some useful stuff:
    - Number of MapBlocks sent
    - Number of FarBlocks sent

- Time out and drop FarBlock meshes that are not rendered at all

- Add a setting for maximum far map send distance on the server

- far_map_allow_generate_to_distance (replace)

- Maybe do something to Z fighting that happens sometimes, or at least explain
  it somehow

- The far map generation limits should be a world-specific setting so that if
  you want to have a large world with a lot of wasted hard drive space, you can
  set it as such, and if not, you don't accidentally bloat it up by leaving some
  global setting enabled

- Allow the client to tell the server how many meshes it wants per second so
  that systems that get choppy framerates with 20 meshes per frame but work fine
  with 5 meshes per frame can be made playable

[paramat]

• Remove auto view range

Reading the forum many players (including me) are choosing to set min and max equal to disable the auto adjust, i feel this won't be missed much.

For a guide to the vertical range farmap should cover, mgv7 can occasionally have mountains up to 500, mgwatershed up to 1000. I guess 'far map generation limits' will have separate horizontal and vertical limits. For some uses (deep space / floatland realms) the vertical range will need to be equal to the horizontal.

[Fixer-007]

Will be nice to have both 'Far map rendering' and 'improved map transfer' in Minetest :)

Make it so that empty FarBlocks don't consume huge amounts of memory

It could be a big problem on 32 bit builds (already susceptible to crashes in big forests in default MTG).

Paramat: Reading the forum many players (including me) are choosing to set min and max equal to disable the auto adjust, i feel this won't be missed much.

I myself use equal min and max when playing on actual multiplayer servers (it gives me some compromise, I'm not perfectly happy with it either).
Auto view range positive sides for me:

• can enlarge view range dramatically in simple env (usually in singleplayer it is all the way to 240).
• lowers view to increase fps in very crowded places (obviously).

Negative sides:

• view range oscillation that is very noticeable in areas near forests when you walk forward (and quite irritating btw).
• looking back at high range will dramatically slow fps (renders huge chunk of previously loaded blocks) and will reduce range fast.

Current auto view range needs rewrite or be non-default.

Increase default target FPS to 60

+1

[VanessaE]

Seems to work okay for me. Some caveats (some of this has been mentioned in IRC, but putting it here "for the record"):

• the farmap blocks are fully lit even at night.
• sometimes it's the inside of a building that comes up as "representative" for a given group of nodes. On one of my maps, for example, a tall concrete building shows as a mixture of concrete and multiple colors of wool in the farmap, due to objects inside being made from significant amounts of wool in various colors. Might be a good idea to give the nodes closest to the player more "weight" when choosing a representative node for the group.
• this may not necessarily be farmap related: after some amount of time, the client starts to lose FPS, and the "detailed" view range (i.e. where it cuts over to farmap) becomes increasingly foreshortened. I noticed this by accident when I joined a server, opened the ESC/Pause menu so I could go over to another window to do something, and left the client sitting there without actually playing for a while (20 or 30 mins). When I closed the Pause menu to start playing, the FPS and view range were much shorter than when I started.
• Sometimes the engine will send me only the farmap, but not the detailed terrain, leaving me just sitting there looking at farmap blocks until I get close enough to "force it", then I can back away and still see the details.
• noclipping through the ground, e.g. when deep underground, shows "terrain" (well, blocks of stone et al) when ordinarily minetest would show just the skybox/void, with black-ish caves and user-dug holes/tunnels. Instead I see somewhat-distant far map blocks with detailed caves/etc. between me and them. This looks very weird, and a bit confusing, in practice. Example:
  

(In this image, I am inside solid ground, 10 or 15 meters back from a void I created via Worldedit. Brightness/contrast adjusted to make it more visible, relative to the inset upper left)

• Under Xubuntu 15.10, open source radeon driver, farmap blocks display more or less as expected (similar to the screenshot above). Under Debian 8.2 with fluxbox window manager, open source radeon driver, farmap is entirely solid white, fully lit blocks - no visible textures. Under Debian 8.2 with Catalyst/fglrx, the farmap seems to work as it does on Xubuntu.

As an aside, all of my servers have this PR in place for testing (far_map_allow_generate is unset/false).

[sofar]

Pulled this just today, only 1 tiny merge conflict (trivial).

Had a blast with farmap range at 2000. I didn't see any issues at all, whatsoever. It's like a giant minimap.

[red-001]

I get about 40 build errors when I try to build using Visual studio 2013. Most of them in atlas.h.

[C1ffisme]

Is this in any way laptop friendly?

Just wondering, I just thought what @sofar 's comment was 👍 worthy. I normally have to keep my render distance at 5 - 25 (because of heat), and this would help.

[Fixer-007]

Can't cross compile it on linux via mingw-w64 for my windows desktop, gives me declaration errors in clientiface.cpp

[lisacvuk]

I just tested it. Specs:
Pentium D 820,
3GB RAM,
GeForce 210 (1GB),
Kubuntu 14.04.
Wanted FPS: 30.
Stable 30 FPS on 500, when I remove wanted FPS it lags about 1000 range. I didn't notice any rendering anomalies.

[celeron55]

I rebased this today. Feel free to check if I broke any recent commits.

[Fixer-007]

Compile error: https://i.imgur.com/Sj8LCcB.png

[celeron55]

@Fixer-007 You could try it now again.

[Fixer-007]

Tried just now, error: https://i.imgur.com/AQSM6Yt.png

[Fixer-007]

Windows users (64bit) can finally try it out!
minetest-0.4.13-089f9bb-win64-far-map-PR3810.zip
Have fun and bug report.

Farmap works. Will test it more.
Also, it looks like this PR reduces/eliminates most of the stutters that plagued some people like me for so long, example: https://imgur.com/UW6TIRD Needs testing in fraps to really verify that, I was subjective.

[paramat]

^ Mgfractal.
A quick 30min test, with farmap range 1024, normal view range 330. FPS for this screenshot was 23 and memory usage levelled out at 2GB.

[Fixer-007]

Looks like there is a memory leak, it starts to eat memory pretty fast until slight freezes begin and in few minutes warning from windows about memory full. My RAM is 8gb. CPU usage is also quite high at 80%. Farmap block loading is not that fast after those changes.

[paramat]

s32 render_range_nodes = getEnv().getClientMap().getMapDrawControl().wanted_range;
s32 render_range_mapblocks_1d = render_range_nodes / MAP_BLOCKSIZE + 1;
s32 render_range_mapblocks_3d = render_range_mapblocks_1d * render_range_mapblocks_1d *
render_range_mapblocks_1d;
s32 limit_required_for_rendering = render_range_mapblocks_3d * 1;

Volume of a sphere is 4/3 * pi * r^3 ~= 4.2 * r^3
The default 5000 limit allows a fully rendered sphere with view range of 170 nodes, not much.
I'm concerned your code would not allow enough mapblocks for a certain view range, it would only increase the limit at 5000 ^ 0.33333 * 16 = 274 nodes view range.

[celeron55]

Oh, looks like it should be something like render_range_nodes * 2 / MAP_BLOCKSIZE + 1 instead of render_range_nodes / MAP_BLOCKSIZE + 1. I thought I did it like that but apparently I didn't.

[celeron55]

4 months ago I split out the map transfer portion of this and forgot to mention here: https://github.com/celeron55/minetest/commits/block_transfer_v2

Feel free to test it; that would be useful. I know it contains the same bugs this branch has though.

Fixing and merging it would probably be wise, because it's a better design than the current one. I am aiming for doing that but don't know about when I'll feel like it.

[paramat]

@celeron55 here's an interesting alternative approach to LOD for MC-like games https://0fps.net/2018/03/03/a-level-of-detail-method-for-blocky-voxels/

[paramat]

I can't see a point keeping this open.

[paramat]

Here are sofar's videos of the farmap feature:
https://www.youtube.com/watch?v=gQ8fo5VBKEM
https://www.youtube.com/watch?v=WLO-Mn4DcOs
https://www.youtube.com/watch?v=tJH6lqLVce4

[LoneWolfHT]

Those links are broken somehow, you have to manually copy them

[paramat]

Thanks, fixed.