Perform server occlusion check before a block is loaded or generated. #14148
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lhofhansl
changed the title
sfan5
reviewed
Dec 22, 2023
|
Done... Now with the faster load times MT gets really useful on large viewing ranges. |
|
Note that when blocks loaded as active blocks in So if a block does not exist (not generated), it will not be generated by the serverenvironment, and the next step will enqueue it again. And each time emerging will check the map, then the DB, and then conclude not to generate the block. I would like to take this pressure off the DB. What do we think? Should active blocks be generated too? And if not, why not? And how to do we deal with repeated enqueues on the emerge queue then? I'd suggest we allow generating, to avoid unnecessary load on the DB, and I think it's what player would want too. |
That sound sensible |
|
Code has my approval 👍 but it'll be a few days until I can test how it actually performs. |
|
For my performance test I used an "open air" scene - so that not too many blocks are hidden - at viewing_range 1000. (I had #14152 applied as well) Generation without this PR took 7 minutes, with PR 3m 41s. It's not too surprising. Before we would load/generate many blocks just to decide that they are not visible in the first place. Now we can avoid loading/generating blocks that we know are not visible. In scenes with more hidden blocks this will be even more pronounced. |
sfan5
approved these changes
Dec 29, 2023
siboehm
added a commit
to Astera-org/minetest
that referenced
this pull request
Jan 17, 2024
Add ZMQ Worlds shittiest minetest interface CI MacOS install zmq Handshakes, plus send Screenshots back cleanup Extract all command names Move remote handler into own file hook mouse . non-working protobuf commit switch to capnproto Send image with capnp Send reward Multi keypress in the client support all possible keys Add environment.yml Commit unmodified eluther gym env Fail to close the inventory Minetester define action and obs space Gymnasium basic work gym env check bounds switch dummy client to pygame More work on gym client more progress First working gym client Rm old client Combine isRemote() and isRandom() Make client port configurable fix camera mode and sneak . fix remote cli flag vscode Add basic reward mod Parse reward from HUD in C++ . Cmake changes to get Linux building README: add linux packages for capnp minor linux changes save png HACK . Linux CI deps tri again tri again gitmodules submodules submodules . First draft env start xvfb more headless fix screenshot startup issue add handshaker for debugging create a new world when starting a game Now `minetest --gameid <foo> --go` should work. Previously it would fail because a world was not specified, and the default one did not exist. Change-Id: Ic6001f73713d59930e96c3f2a30395f5b0b8cbb9 Improve Irrlicht instructions (minetest#14055) Specify the version when cloning. Move instructions from CMakeLists to docs/compiling/. Check if liquid can flow into empty node before picking it as source (minetest#14057) Fix unittest failure for release versions (minetest#14067) GUIFormspecMenu: Fix race condition between quit event and cleanup in Game (minetest#14010) To not instantly free GUIFormSpec upon close/quit, Game periodically cleans up the remaining instance on the next frame. When a new formspec is received and processed after closing the previous formspec but before the cleanup in Game, the formspec would be closed regardless. This now re-creates the formspec when the old one is already pending for removal. Add sound volume when unfocused setting (minetest#14083) This adds a new setting to set sound volume multiplier when Minetest window is unfocused/inactive (sound_volume_unfocused, located in Settings > Graphics and Audio > Audio > Volume when unfocused). If the window is not focused, the sound volume will be multiplied by sound_volume_unfocused setting. The sound volume will be set back to sound_volume again when the window is focused. Avoid movement jitter (minetest#13093) This allows the client and server to agree on the position of objects and attached players even when there is lag. Try to fix safeWriteToFile producing empty files on Windows (minetest#14085) Use win32 APIs to write the temporary file before copying to the final destination. Because we've observed the final file being empty, we suspect that std::ostream::flush is not flushing. Also add a test for it. Remove use_texture_alpha compatibility code for nodeboxes & meshes (minetest#13929) Warning: inform about entity name when bug detected about attachement (minetest#13354) Clean up porting.h a bit Improve clock_gettime usage - correctly use value of _POSIX_MONOTONIC_CLOCK - drop special path for macOS: it supports clock_gettime since macOS 10.12 Reduce test framework macrosity Delete clang-format files and comments (minetest#14079) Fix Windows architecture reporting in sysinfo Upload artifacts in MinGW CI Get rid of VERSION_EXTRA for buildbot This is probably a leftover of when CMake didn't automatically detect the revision from git. Hash-check buildbot dependencies Inventory: prevent item loss when stacking oversized ItemStacks (minetest#14072) Remove usage of removed "PP" macro This fixes a compilation error introduced by e7be135. Allow running individual benchmarks mirrors and reuses the option from 2f6a9d1 Try to benchmark common MapBlock usage Allocate data seperately from MapBlock class again This effectively reverts commit b3503e7. Change MapBlock content cache to a vector Get rid of parent pointer in MapBlock Elide MapBlock::contents_cached Reduce size of some MapBlock members Also adds assertions to catch refcounting errors (on a debug build). Reorder members of MapBlock for performance Before and after as obtained via `pahole -C MapBlock bin/minetest`: /* size: 336, cachelines: 6, members: 23 */ /* sum members: 329, holes: 4, sum holes: 7 */ vs. /* size: 336, cachelines: 6, members: 23 */ /* sum members: 329, holes: 2, sum holes: 7 */ There is not much to be gained by packing but I made sure to move the most important data (mainly for the client) into the first cache line. Update porting.h to fix build errors on macOS 14 / Xcode 15 Allow cheaper culling checks at a distance (minetest#14073) * Allow cheaper culling checks at a distance * Pick a random ray, so that far missing block will eventually be shown Extract Game::drawScene from Game::updateFrame Add `touch_controls` boolean to `get_player_window_information()` (minetest#14092) MinGW toolchain refresh Add dithering (minetest#9014) Android: Pause rendering while the app is paused (minetest#14058) Hand roll UTF-16 conversion in CGUITTFont (minetest#14121) Extend bone override capabilities (minetest#12388) Fix touch input on Linux The code relied on touch IDs being consecutive. This is true on Android, but not on Linux. Therefore, touch input on Linux was broken since 53886dc. Enable segment heap on Windows Fix TouchScreenGUI ignoring server-sent pitch changes Manually configurable minimum protocol version (minetest#14054) Partially address minetest#13483. Server operators can set a minimum protocol version to match the game requirements (or any other restriction they may want), and it's applied as an additional constraint on top of the baseline compatibility range, optional strict_protocol_version_checking, and any kick-on-join used by the game/mods. Fix on_(grant|revoke) not being run by mods Split windows from linux CI workflows Fix set_bone_position regression (error on passing none) Initial implementation of 'Godrays' Make volumetric light effect strength server controllable - Make volumetric light effect strength server controllable - Separate volumetric and bloom shader pipeline - Require bloom to be enable, scale godrays with bloom Touchscreen: Make server-sent overrides of button textures work (minetest#14145) Support specifying game in config In minetest.conf: ``` game_dir = /path/to/game ``` Change-Id: I790cc6ea91aa988e82e0a1fda23ab4ae2f8026a3 Link with -latomic Rework server stepping and dtime calculation Address some clang-tidy warnings Update clang-tidy workflow Fix minor issue with log_deprecated() Remove non-existent textures from texture_packs.md These textures were removed 5 years ago by 326eeca. Make the loading screen progress bar respect "gui_scaling" MacOS: Add codesigning instructions to docs (minetest#14060) Optimize and improve built-in PNG writer (minetest#14020) Update CMakeLists.txt to fix MacOS build (minetest#14160) Co-authored-by: sfan5 <sfan5@live.de> Support both mouse and touch input in GUIs in a single binary (minetest#14146) Avoid short overflow with large viewing ranges (minetest#14175) Rename `hud_elem_type` to `type` (minetest#14065) Fix AsyncRunStep() skipping steps when dtime < 1 ms Clean up OS-specific initialization Clean up gettext initialization Enable some runtime hardening on win32 Perform server occlusion check before a block is loaded or generated (minetest#14148) Do not emerge blocks in the active_object_send_range_blocks range (minetest#14152) The active object range is about active objects (not blocks). Activate blocks (and hence any object "in" them) in the cone define by the active object range (and fov) when they are loaded (i.e. visible), otherwise ignore them. Extend sanity checks in ActiveBlockList::update also fixes the space indentation Add missing header for gcc-14 https://gcc.gnu.org/gcc-14/porting_to.html Signed-off-by: Alfred Wingate <parona@protonmail.com> Method add_pos for object/player (minetest#14126) Comply with base64 license terms (minetest#14199) Remove reference to defunct gitlab docker image see minetest#14164 Fix tonemapping effect Apply saturation even if tonemapping is disabled Legible Lua profiler (minetest#14142) Replace clientmap's MeshBufListList with a hashmap Use AL_SOFT_direct_channels_remix extension for non-positional stereo sounds (minetest#14195) Don't apply gui_scaling & DPI twice to table[] / textlist[] scrollbar (minetest#14206) Remove server's address and port from pause menu (minetest#14082) Touchscreen: Recognize double-taps as double-clicks (minetest#14187) Fix GameUI text staying visible during shutdown. (minetest#14197) Don't run CDB update_detector more than once (minetest#14214) Remove controls listed in the pause menu (no touchscreen) (minetest#13282) Add "--needed" to Arch command to avoid reinstalling packages Fix logic in porting::attachOrCreateConsole() No functional change but now the comment is actually correct. Add unittest to check thread_local destructor brokenness Avoid unused argument spam with MinGW-clang Fix native thread handle usage on win32 Fix some console window behavior on Windows Ensure deterministic client occlusion culling and minor improvements (minetest#14212) * Ensure deterministic client occlusion culling * Increase culling optimize distance slightly * More accurate culling when sampling Android: Add selection dialog (drop down/combo box) (minetest#13814) - The handling of IGUIComboBox uses the new setAndSendSelected() method. - getDialogState() is now getInputDialogState() and returns the state of the input dialog. - getLastDialogType() is added and returns current/last shown dialog's type. - getInputDialogState() now returns an enum instead of int. - getAndroidUIInput() now returns void instead of bool. - New data types (enum) are added: (1) GameActivity.DialogType (Java) and porting::AndroidDialogType (C++) (2) GameActivity.DialogState (Java) and porting::AndroidDialogState (C++) - When showing a text input dialog, there is no custom accept button text any more. - showDialog()/showDialogUI() for text input is now showTextInputDialog()/showTextInputDialogUI(). - showInputDialog()/showDialogUI() for text input is now showTextInputDialog()/showTextInputDialogUI(). - getDialogValue()/getInputDialogValue() is now getDialogMessage()/getInputDialogMessage(). Co-authored-by: Gregor Parzefall <82708541+grorp@users.noreply.github.com> Extend capabilities of Address class Some minor cleanups for UDPSocket class Rework client connecting and enable fallback address use Fix dividing by zero crashes in texture modifiers slimmer gitignore Bit of cleanup ruff First ZMQ Req returns image . add a basic env test Document env setup ci ci ci ci . ci ci ci ci ci make build a bit faster maybe ci ci ci ci pytest pytest again add channels pytest forgot to setup env cleanup minetest env pytest timeout & logging . doc . pytest Rm wrong comment & stdout printing ci ci Better macos instructions rm android and windows ci rm docker ci try fix macos ci try fix clang tidy ci macos submodules macos submodules macos submodules 1 macos submodules 1 macos submodules 1 macos submodules 1 macos submodules 1 Add test world macos submodules 11 macos build again move reward mod move world . no mdofi
siboehm
added a commit
to Astera-org/minetest
that referenced
this pull request
Jan 17, 2024
Add ZMQ Worlds shittiest minetest interface CI MacOS install zmq Handshakes, plus send Screenshots back cleanup Extract all command names Move remote handler into own file hook mouse . non-working protobuf commit switch to capnproto Send image with capnp Send reward Multi keypress in the client support all possible keys Add environment.yml Commit unmodified eluther gym env Fail to close the inventory Minetester define action and obs space Gymnasium basic work gym env check bounds switch dummy client to pygame More work on gym client more progress First working gym client Rm old client Combine isRemote() and isRandom() Make client port configurable fix camera mode and sneak . fix remote cli flag vscode Add basic reward mod Parse reward from HUD in C++ . Cmake changes to get Linux building README: add linux packages for capnp minor linux changes save png HACK . Linux CI deps tri again tri again gitmodules submodules submodules . First draft env start xvfb more headless fix screenshot startup issue add handshaker for debugging create a new world when starting a game Now `minetest --gameid <foo> --go` should work. Previously it would fail because a world was not specified, and the default one did not exist. Change-Id: Ic6001f73713d59930e96c3f2a30395f5b0b8cbb9 Improve Irrlicht instructions (minetest#14055) Specify the version when cloning. Move instructions from CMakeLists to docs/compiling/. Check if liquid can flow into empty node before picking it as source (minetest#14057) Fix unittest failure for release versions (minetest#14067) GUIFormspecMenu: Fix race condition between quit event and cleanup in Game (minetest#14010) To not instantly free GUIFormSpec upon close/quit, Game periodically cleans up the remaining instance on the next frame. When a new formspec is received and processed after closing the previous formspec but before the cleanup in Game, the formspec would be closed regardless. This now re-creates the formspec when the old one is already pending for removal. Add sound volume when unfocused setting (minetest#14083) This adds a new setting to set sound volume multiplier when Minetest window is unfocused/inactive (sound_volume_unfocused, located in Settings > Graphics and Audio > Audio > Volume when unfocused). If the window is not focused, the sound volume will be multiplied by sound_volume_unfocused setting. The sound volume will be set back to sound_volume again when the window is focused. Avoid movement jitter (minetest#13093) This allows the client and server to agree on the position of objects and attached players even when there is lag. Try to fix safeWriteToFile producing empty files on Windows (minetest#14085) Use win32 APIs to write the temporary file before copying to the final destination. Because we've observed the final file being empty, we suspect that std::ostream::flush is not flushing. Also add a test for it. Remove use_texture_alpha compatibility code for nodeboxes & meshes (minetest#13929) Warning: inform about entity name when bug detected about attachement (minetest#13354) Clean up porting.h a bit Improve clock_gettime usage - correctly use value of _POSIX_MONOTONIC_CLOCK - drop special path for macOS: it supports clock_gettime since macOS 10.12 Reduce test framework macrosity Delete clang-format files and comments (minetest#14079) Fix Windows architecture reporting in sysinfo Upload artifacts in MinGW CI Get rid of VERSION_EXTRA for buildbot This is probably a leftover of when CMake didn't automatically detect the revision from git. Hash-check buildbot dependencies Inventory: prevent item loss when stacking oversized ItemStacks (minetest#14072) Remove usage of removed "PP" macro This fixes a compilation error introduced by e7be135. Allow running individual benchmarks mirrors and reuses the option from 2f6a9d1 Try to benchmark common MapBlock usage Allocate data seperately from MapBlock class again This effectively reverts commit b3503e7. Change MapBlock content cache to a vector Get rid of parent pointer in MapBlock Elide MapBlock::contents_cached Reduce size of some MapBlock members Also adds assertions to catch refcounting errors (on a debug build). Reorder members of MapBlock for performance Before and after as obtained via `pahole -C MapBlock bin/minetest`: /* size: 336, cachelines: 6, members: 23 */ /* sum members: 329, holes: 4, sum holes: 7 */ vs. /* size: 336, cachelines: 6, members: 23 */ /* sum members: 329, holes: 2, sum holes: 7 */ There is not much to be gained by packing but I made sure to move the most important data (mainly for the client) into the first cache line. Update porting.h to fix build errors on macOS 14 / Xcode 15 Allow cheaper culling checks at a distance (minetest#14073) * Allow cheaper culling checks at a distance * Pick a random ray, so that far missing block will eventually be shown Extract Game::drawScene from Game::updateFrame Add `touch_controls` boolean to `get_player_window_information()` (minetest#14092) MinGW toolchain refresh Add dithering (minetest#9014) Android: Pause rendering while the app is paused (minetest#14058) Hand roll UTF-16 conversion in CGUITTFont (minetest#14121) Extend bone override capabilities (minetest#12388) Fix touch input on Linux The code relied on touch IDs being consecutive. This is true on Android, but not on Linux. Therefore, touch input on Linux was broken since 53886dc. Enable segment heap on Windows Fix TouchScreenGUI ignoring server-sent pitch changes Manually configurable minimum protocol version (minetest#14054) Partially address minetest#13483. Server operators can set a minimum protocol version to match the game requirements (or any other restriction they may want), and it's applied as an additional constraint on top of the baseline compatibility range, optional strict_protocol_version_checking, and any kick-on-join used by the game/mods. Fix on_(grant|revoke) not being run by mods Split windows from linux CI workflows Fix set_bone_position regression (error on passing none) Initial implementation of 'Godrays' Make volumetric light effect strength server controllable - Make volumetric light effect strength server controllable - Separate volumetric and bloom shader pipeline - Require bloom to be enable, scale godrays with bloom Touchscreen: Make server-sent overrides of button textures work (minetest#14145) Support specifying game in config In minetest.conf: ``` game_dir = /path/to/game ``` Change-Id: I790cc6ea91aa988e82e0a1fda23ab4ae2f8026a3 Link with -latomic Rework server stepping and dtime calculation Address some clang-tidy warnings Update clang-tidy workflow Fix minor issue with log_deprecated() Remove non-existent textures from texture_packs.md These textures were removed 5 years ago by 326eeca. Make the loading screen progress bar respect "gui_scaling" MacOS: Add codesigning instructions to docs (minetest#14060) Optimize and improve built-in PNG writer (minetest#14020) Update CMakeLists.txt to fix MacOS build (minetest#14160) Co-authored-by: sfan5 <sfan5@live.de> Support both mouse and touch input in GUIs in a single binary (minetest#14146) Avoid short overflow with large viewing ranges (minetest#14175) Rename `hud_elem_type` to `type` (minetest#14065) Fix AsyncRunStep() skipping steps when dtime < 1 ms Clean up OS-specific initialization Clean up gettext initialization Enable some runtime hardening on win32 Perform server occlusion check before a block is loaded or generated (minetest#14148) Do not emerge blocks in the active_object_send_range_blocks range (minetest#14152) The active object range is about active objects (not blocks). Activate blocks (and hence any object "in" them) in the cone define by the active object range (and fov) when they are loaded (i.e. visible), otherwise ignore them. Extend sanity checks in ActiveBlockList::update also fixes the space indentation Add missing header for gcc-14 https://gcc.gnu.org/gcc-14/porting_to.html Signed-off-by: Alfred Wingate <parona@protonmail.com> Method add_pos for object/player (minetest#14126) Comply with base64 license terms (minetest#14199) Remove reference to defunct gitlab docker image see minetest#14164 Fix tonemapping effect Apply saturation even if tonemapping is disabled Legible Lua profiler (minetest#14142) Replace clientmap's MeshBufListList with a hashmap Use AL_SOFT_direct_channels_remix extension for non-positional stereo sounds (minetest#14195) Don't apply gui_scaling & DPI twice to table[] / textlist[] scrollbar (minetest#14206) Remove server's address and port from pause menu (minetest#14082) Touchscreen: Recognize double-taps as double-clicks (minetest#14187) Fix GameUI text staying visible during shutdown. (minetest#14197) Don't run CDB update_detector more than once (minetest#14214) Remove controls listed in the pause menu (no touchscreen) (minetest#13282) Add "--needed" to Arch command to avoid reinstalling packages Fix logic in porting::attachOrCreateConsole() No functional change but now the comment is actually correct. Add unittest to check thread_local destructor brokenness Avoid unused argument spam with MinGW-clang Fix native thread handle usage on win32 Fix some console window behavior on Windows Ensure deterministic client occlusion culling and minor improvements (minetest#14212) * Ensure deterministic client occlusion culling * Increase culling optimize distance slightly * More accurate culling when sampling Android: Add selection dialog (drop down/combo box) (minetest#13814) - The handling of IGUIComboBox uses the new setAndSendSelected() method. - getDialogState() is now getInputDialogState() and returns the state of the input dialog. - getLastDialogType() is added and returns current/last shown dialog's type. - getInputDialogState() now returns an enum instead of int. - getAndroidUIInput() now returns void instead of bool. - New data types (enum) are added: (1) GameActivity.DialogType (Java) and porting::AndroidDialogType (C++) (2) GameActivity.DialogState (Java) and porting::AndroidDialogState (C++) - When showing a text input dialog, there is no custom accept button text any more. - showDialog()/showDialogUI() for text input is now showTextInputDialog()/showTextInputDialogUI(). - showInputDialog()/showDialogUI() for text input is now showTextInputDialog()/showTextInputDialogUI(). - getDialogValue()/getInputDialogValue() is now getDialogMessage()/getInputDialogMessage(). Co-authored-by: Gregor Parzefall <82708541+grorp@users.noreply.github.com> Extend capabilities of Address class Some minor cleanups for UDPSocket class Rework client connecting and enable fallback address use Fix dividing by zero crashes in texture modifiers slimmer gitignore Bit of cleanup ruff First ZMQ Req returns image . add a basic env test Document env setup ci ci ci ci . ci ci ci ci ci make build a bit faster maybe ci ci ci ci pytest pytest again add channels pytest forgot to setup env cleanup minetest env pytest timeout & logging . doc . pytest Rm wrong comment & stdout printing ci ci Better macos instructions rm android and windows ci rm docker ci try fix macos ci try fix clang tidy ci macos submodules macos submodules macos submodules 1 macos submodules 1 macos submodules 1 macos submodules 1 macos submodules 1 Add test world macos submodules 11 macos build again move reward mod move world . no mdofi
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As I looked into loading times for large viewing ranges, it became obvious that loading blocks from the DB (including deserialization) as well as generating non-existent blocks is the limiting factor.
It turns out that in order to do server-side occlusion checks we do not actually need the block loaded.
So what this is doing is to occlusion-cull blocks before they are loaded and do not load them at all if they not visible in the first place.
I easily get a 3x improvement in loading times at
viewing_range= 1000. And about a 2x improvement from generate times.(20s as opposed a minute)
Goal of the PR
Reduce map loading time.
How does the PR work?
Occlusion check blocks before they are loaded, and avoid loading them if they are not visible.
Does it resolve any reported issue?
No directly.
Does this relate to a goal in the roadmap?
Perhaps. Larger viewing ranging is a goal, no?
If not a bug fix, why is this PR needed? What usecases does it solve?
Significantly improve map loading time. Significantly reduce database load
To do
This PR is Ready for Review.
How to test
viewing_range= 1000max_block_send_distance= 64max_block_generate_distance= 64client_mapblock_limit= 300000max_simultaneous_block_sends_per_client= 160emergequeue_limit_diskonly= 256emergequeue_limit_generate= 256Note the times it takes to load the full map with and without the PR. The set of blocks displayed should be the same.