Mimalloc Benchmarks

[nick-l-o3de]

mimalloc issues

Spew on startup

  --   AzCore uses mimalloc v2.2.7 (MIT) from https://github.com/microsoft/mimalloc.git
--   Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.49.0.windows.1")
--
--   No build type selected, default to 'Release'
--   Note: when building with Visual Studio the build type is specified when building.
--   For example: 'cmake --build . --config=Release
--   Architecture: x64
--   Skip collecting memory on program exit (MI_SKIP_COLLECT_ON_EXIT=ON)
--   Use the C++ compiler to compile (MI_USE_CXX=ON)
--
--   Library name     : mimalloc
--   Version          : 2.2.7
--   Build type       : release
--   C++ Compiler     : C:/Program Files/Microsoft Visual Studio/18/Community/VC/Tools/MSVC/14.50.35717/bin/Hostx64/x64/cl.exe
--   Compiler flags   : /Zc:__cplusplus
--   Compiler defines : MI_SKIP_COLLECT_ON_EXIT=1;MI_CMAKE_BUILD_TYPE=release;MI_BUILD_RELEASE
--   Link libraries   : psapi;shell32;user32;advapi32;bcrypt
--   Build targets    : static
--

mimalloc-debug.lib is produced when in profile mode.

Crash unloading and loading levels in mimalloc+hpha mode

>	[Inline Frame] Atom_RHI.Public.dll!AZStd::deque<AZ::RHI::PoolAllocator::Garbage,AZStd::allocator,2,8>::emplace_back(unsigned int &) Line 825	C++
 	[Inline Frame] Atom_RHI.Public.dll!std::queue<AZ::RHI::PoolAllocator::Garbage,AZStd::deque<AZ::RHI::PoolAllocator::Garbage,AZStd::allocator,2,8>>::emplace(unsigned int &) Line 140	C++
 	Atom_RHI.Public.dll!AZ::RHI::PoolAllocator::DeAllocate(AZ::RHI::VirtualAddress allocation) Line 89	C++
 	[Inline Frame] Atom_RHI_DX12.Private.dll!AZ::RHI::MemorySubAllocator<AZ::RHI::MemorySubAllocatorTraits<ID3D12Resource,AZ::DX12::MemoryPageAllocator,AZ::RHI::PoolAllocator>>::DeAllocate(const AZ::RHI::MemoryAllocation<ID3D12Resource> &) Line 156	C++
 	Atom_RHI_DX12.Private.dll!AZ::DX12::ShaderResourceGroupPool::ShutdownResourceInternal(AZ::RHI::DeviceResource & resourceBase) Line 172	C++
 	Atom_RHI.Public.dll!AZ::RHI::DeviceResourcePool::ShutdownResource(AZ::RHI::DeviceResource * resource) Line 203	C++
 	Atom_RHI.Public.dll!AZ::RHI::DeviceResource::Shutdown() Line 126	C++
 	Atom_RHI.Public.dll!AZ::RHI::Object::release() Line 59	C++

The innards of "this" seemed fine, but the innards of "this->m_garbage" were all null.

---

Timings

• All timings are taken four times.
• The first timing is discarded to make sure cache is warm so 'first run' is the first non-discarded run.
• All runs are done immediately after each other, without issuing any other commands in between or changing the env.
• All timings are using multiplayer sample, which has a lot of assets, work to do loading levels, etc.
• Timings are in seconds.

Names of runs:

• MIMALLOC+HPHA -DO3DE_OS_ALLOCATOR_USE_MIMALLOC=ON -DO3DE_SYSTEM_ALLOCATOR_USE_MALLOC=OFF (mimalloc and HPHA)
• MIMALLOC: -DO3DE_OS_ALLOCATOR_USE_MIMALLOC=ON -DO3DE_SYSTEM_ALLOCATOR_USE_MALLOC=ON (Mimalloc, no HPHA)
• HPHA -DO3DE_OS_ALLOCATOR_USE_MIMALLOC=OFF -DO3DE_SYSTEM_ALLOCATOR_USE_MALLOC=OFF (o3de vanilla, with HPHA)
• MALLOC -DO3DE_OS_ALLOCATOR_USE_MIMALLOC=OFF -DO3DE_SYSTEM_ALLOCATOR_USE_MALLOC=ON (o3de vanilla, no HPHA)

rmdir /s /q Cache && AssetProcessorBatch.exe --zeroAnalysisMode

LOWER IS BETTER

Run	MIMALLOC+HPHA	MIMALLOC	HPHA	MALLOC
First run:	679.5s	681.4s	668.0s	654.1s
Second run:	642.5s	646.1s	670.3s	650.0s
Third run:	646.3s	664.7s	694.9s	649.1s
-------------	---------------	---------------	---------------	----------------
Average	656.1s	664.6s	677.3s	651.1s (best)
Variance	18.5s (2.8%)	18.5s (2.7%)	13.4s (1.9%)	2.5s (0.4%)
Diff v best	+5.0s (<1%>)	13.5s (2.0%)	26.2s (3.8%)	(is best)

Best to worst: MALLOC, MIMALLOC+HPHA, MIMALLOC, HPHA

Editor load/run/start/quit test

note that Asset Processor is running and idle in the background so no asset processing happens and all assets
are already ready during this test.

Measure-Command -Expression { .\build\windows\bin\profile\Editor.exe --autotest_mode --skipWelcomeScreenDialog --runpythontest d:\o3de-repos\o3de-multiplayersample\Assets\runtest.py --pythontestcase foo | Out-Default ; $exitCode = $LASTEXITCODE} ; Write-Host "Exit code: $exitCode"

With "runtest.py"

import os, sys
import azlmbr.legacy.general as general
import azlmbr.framework as framework

# 30 frames is enough for all texture and model streaming to complete.
# note that this will add a consistent amount of time to each test, but, 
# since most of those 30 frames are used up cpu/gpu bound loading stuff into memory, then
# if some memory allocation strategy is faster, those frames will complete FASTER
# and thus shorten the time of the test.

waitframes = 30 
repeats = 10

# disable icons/helpers
general.idle_enable(True)
general.idle_wait_frames(1)
helpers_visible = general.is_helpers_shown()
icons_visible = general.is_icons_shown()

if helpers_visible:
    general.toggle_helpers()
if icons_visible:
    general.toggle_icons()

# load a level and wait a frame
general.idle_wait_frames(1)
general.open_level_no_prompt("NewStarbase")


for i in range(0, repeats):
    general.idle_wait_frames(waitframes)
    general.enter_game_mode()
    general.idle_wait_frames(waitframes)
    general.exit_game_mode()

general.idle_wait_frames(1)
framework.Terminate(0)

LOWER IS BETTER

Run	MIMALLOC+HPHA	MIMALLOC	HPHA	MALLOC
1	195.4s	171.6s	182.2s	181.1s
2	184.6s	177.7s	184.9s	185.5s
3	182.1s	172.5s	194.3s	183.4s
4	189.8s	177.5s	192.9s	174.1s
5	196.3s	173.0s	199.0s	181.5s
6	177.8s	170.9s	189.4s	176.6s
-------------	---------------	---------------	---------------	----------------
Average	187.7s	173.9s (BEST)	190.5s	180.4s
Variance	~9.2s	~3.4s	~8.4s	~ 4.5s
Diff v best	13.8s slower	*	16.6s slower	6.5s slower

• Note that MIMALLOC+HPHA crashed 2x during test, results discarded, test restarted.
• Will investigate later.

High memory watermark

I start the game running in the editor, which involves running the server, too, and check the
maximum allocated amount of memory.

Note that this value was identical across multiple runs, so only one is shown.

LOWER IS BETTER

Run	MIMALLOC+HPHA	MIMALLOC	HPHA	MALLOC
1	13.7gb	12.5gb	10.8gb	10.1gb (BEST)
-------------	---------------	---------------	---------------	----------------
Diff v best	3.6gb	2.4gb	0.7gb	-

FPS

This is the average ms per frame, inside the game runtime in the editor, when running
in an unlimited framerate mode (no vsync). Note that each time I ctrlg/exit, the
ms/f goes up, so this is just the first run (will investigate)

LOWER IS BETTER

Run	MIMALLOC+HPHA	MIMALLOC	HPHA	MALLOC
1	10.4ms	10.7ms	11.7ms	12.6ms
-------------	---------------	---------------	---------------	----------------
Diff v best