Hitting non-empty VmaDeviceMemoryBlock assert

[rumblehhh]

We're hitting the metadata->IsEmpty() assert on shutdown inside VmaDeviceMemoryBlock::Destroy however VmaBlockMetadata_TLSF::DebugLogAllAllocations() isn't printing a list of unfreed allocations - we've defined VMA_DEBUG_LOG.
I see that VmaBlockMetadata_TLSF::DebugLogAllAllocation is getting skipped due to the !block->IsFree() check.

I've copied the VMA specific part of our logs below. The number of allocations seems to line up perfectly with the number of allocations freed. Are we missing anything obvious?

Note that I modified the allocation/freed VMA_DEBUG_LOGs to include the allocation handle to help debug this but all seems fine.

We've (PVRCarbon: https://developer.imaginationtech.com/pvrcarbon/) been using v3.0.0 since March (Vma in general a lot longer) and we've not hit this issue before. I also tried v3.0.1 this morning and see the same behaviour.

It looks like its a particular path is being taken to cause this. Have you got any suggestions?

vmaCreateAllocator
vmaCreateAllocator
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=7680
Created new block #0 Size=33554432, allocation=140737127085760
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=768
Returned from existing block #0, allocation: 140737127085840
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=398272
Returned from existing block #0, allocation: 140737127085920
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=120348
Returned from existing block #0, allocation: 140737127086000
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=120348
Returned from existing block #0, allocation: 140737127086080
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=2368
Returned from existing block #0, allocation: 140737127086160
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=432
Returned from existing block #0, allocation: 140737127086240
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=432
Returned from existing block #0, allocation: 140737127086320
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=35552
Returned from existing block #0, allocation: 140737127086400
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4932
Returned from existing block #0, allocation: 140737127086480
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4932
Returned from existing block #0, allocation: 140737127086560
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=35616
Returned from existing block #0, allocation: 140737127086640
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4932
Returned from existing block #0, allocation: 140737127086720
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4932
Returned from existing block #0, allocation: 140737127086800
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=56800
Returned from existing block #0, allocation: 140737127086880
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=16752
Returned from existing block #0, allocation: 140737127086960
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=16752
Returned from existing block #0, allocation: 140737127087040
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=768
Returned from existing block #0, allocation: 140737127087120
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=72
Returned from existing block #0, allocation: 140737127087200
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=72
Returned from existing block #0, allocation: 140737127087280
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=5120
Returned from existing block #0, allocation: 140737127087360
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1368
Returned from existing block #0, allocation: 140737127087440
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1368
Returned from existing block #0, allocation: 140737127087520
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=5120
Returned from existing block #0, allocation: 140737127087600
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1368
Returned from existing block #0, allocation: 140737127087680
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1368
Returned from existing block #0, allocation: 140737127087760
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=40320
Returned from existing block #0, allocation: 140737127087840
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4188
Returned from existing block #0, allocation: 140737127087920
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4188
Returned from existing block #0, allocation: 140737127088000
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4194304
Returned from existing block #0, allocation: 140737127088080
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1048576
Returned from existing block #0, allocation: 140737127088160
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=262144
Returned from existing block #0, allocation: 140737127088240
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=65536
Returned from existing block #0, allocation: 140737127088320
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=16384
Returned from existing block #0, allocation: 140737127088400
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4096
Returned from existing block #0, allocation: 140737127088480
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1024
Returned from existing block #0, allocation: 140737127088560
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=256
Returned from existing block #0, allocation: 140737127088640
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=64
Returned from existing block #0, allocation: 140737127088720
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=16
Returned from existing block #0, allocation: 140737127088800
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4
Returned from existing block #0, allocation: 140737127088880
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1048576
Returned from existing block #0, allocation: 140737127088960
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=262144
Returned from existing block #0, allocation: 140737127089040
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=65536
Returned from existing block #0, allocation: 140737127089120
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=16384
Returned from existing block #0, allocation: 140737127089200
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4096
Returned from existing block #0, allocation: 140737127089280
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1024
Returned from existing block #0, allocation: 140737127089360
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=256
Returned from existing block #0, allocation: 140737127089440
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=64
Returned from existing block #0, allocation: 140737127089520
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=16
Returned from existing block #0, allocation: 140737127089600
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4
Returned from existing block #0, allocation: 140737127089680
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=2097152
Returned from existing block #0, allocation: 140737127089760
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=524288
Returned from existing block #0, allocation: 140737127089840
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=131072
Returned from existing block #0, allocation: 140737127089920
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=32768
Returned from existing block #0, allocation: 140737127090000
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=8192
Returned from existing block #0, allocation: 140737127090080
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=2048
Returned from existing block #0, allocation: 140737127090160
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=512
Returned from existing block #0, allocation: 140737127090240
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=128
Returned from existing block #0, allocation: 140737127090320
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=32
Returned from existing block #0, allocation: 140737127090400
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=8
Returned from existing block #0, allocation: 140737127090480
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4
Returned from existing block #0, allocation: 140737127090560
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=16384
Returned from existing block #0, allocation: 140737127090640
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=16384
Returned from existing block #0, allocation: 140737127090720
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=356088
Returned from existing block #0, allocation: 140737127090800
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=88800
Returned from existing block #0, allocation: 140737127090880
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=22000
Returned from existing block #0, allocation: 140737127090960
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=5400
Returned from existing block #0, allocation: 140737127091040
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=1300
Returned from existing block #0, allocation: 140737127091120
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=288
Returned from existing block #0, allocation: 140737127091200
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=72
Returned from existing block #0, allocation: 140737127091280
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=12
Returned from existing block #0, allocation: 140737127091360
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4
Returned from existing block #0, allocation: 140737127091440
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=131072
Returned from existing block #0, allocation: 140737127091520
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=20
Returned from existing block #0, allocation: 140737127091600
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=20
Returned from existing block #0, allocation: 140737127091680
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=20
Returned from existing block #0, allocation: 140737127091760
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090800
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090880
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090960
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091040
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091120
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091200
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091280
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091360
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091440
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091520
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=196596
Returned from existing block #0, allocation: 140737127091520
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127085920
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086080
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086160
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086320
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086400
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086560
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086640
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086800
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086880
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087040
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087120
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087280
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087360
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087520
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087600
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087760
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087840
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088000
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088080
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088160
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088240
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088320
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088400
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088480
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088560
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088640
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088720
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088800
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088880
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127088960
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089040
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089120
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089200
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089280
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089360
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089440
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089520
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089600
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089680
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089760
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089840
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127089920
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090000
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090080
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090160
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090240
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090320
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090400
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090480
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090560
vmaAllocateMemoryForBuffer
AllocateMemory: MemoryTypeIndex=0, AllocationCount=1, Size=4587520
Returned from existing block #0, allocation: 140737127090560
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090560
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091680
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091760
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127085760
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127085840
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091520
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127091600
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090640
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127090720
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086000
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086240
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086480
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086720
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127086960
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087200
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087440
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087680
vmaFreeMemory
Freed from MemoryTypeIndex=0, allocation: 140737127087920
vmaDestroyAllocator

[medranSolus]

Hello and thanks for reporting that!
Could you provide us with JSON containing allocator statistics about current memory usage during that situation? You can obtain it via function vmaBuildStatsString() with detailedMap parameter set to true (just call it before destruction of allocator). Also if you would specify which device block is causing said problem that would help us greatly in finding the reason behind all of this (value of m_Id of VmaDeviceMemoryBlock).
Marek Machliński, Developer Technology Engineer Intern, AMD

[rumblehhh]

Hi Marek,
Please see the following json stats.
m_Id is 0 as it seems we only use a single block.

{ "General": { "API": "Vulkan", "apiVersion": "1.3.0", "GPU": "SwiftShader Device (LLVM 10.0.0)", "deviceType": 4, "maxMemoryAllocationCount": 4096, "bufferImageGranularity": 4096, "nonCoherentAtomSize": 256, "memoryHeapCount": 1, "memoryTypeCount": 1 }, "Total": { "BlockCount": 1, "BlockBytes": 33554432, "AllocationCount": 0, "AllocationBytes": 0, "UnusedRangeCount": 4, "UnusedRangeSizeMin": 7776, "UnusedRangeSizeMax": 32706928 }, "MemoryInfo": { "Heap 0": { "Flags": ["DEVICE_LOCAL"], "Size": 2147483648, "Budget": { "BudgetBytes": 1717986918, "UsageBytes": 33554432 }, "Stats": { "BlockCount": 1, "BlockBytes": 33554432, "AllocationCount": 0, "AllocationBytes": 0, "UnusedRangeCount": 4, "UnusedRangeSizeMin": 7776, "UnusedRangeSizeMax": 32706928 }, "MemoryPools": { "Type 0": { "Flags": ["DEVICE_LOCAL", "HOST_VISIBLE", "HOST_COHERENT", "HOST_CACHED"], "Stats": { "BlockCount": 1, "BlockBytes": 33554432, "AllocationCount": 0, "AllocationBytes": 0, "UnusedRangeCount": 4, "UnusedRangeSizeMin": 7776, "UnusedRangeSizeMax": 32706928 } } } } }, "DefaultPools": { "Type 0": { "PreferredBlockSize": 268435456, "Blocks": { "0": { "MapRefCount": 0, "TotalBytes": 33554432, "UnusedBytes": 33554432, "Allocations": 0, "UnusedRanges": 4, "Suballocations": [ {"Offset": 0, "Type": "FREE", "Size": 7776}, {"Offset": 7776, "Type": "FREE", "Size": 831840}, {"Offset": 839616, "Type": "FREE", "Size": 7888}, {"Offset": 847504, "Type": "FREE", "Size": 32706928} ] } }, "DedicatedAllocations": [ ] } }, "CustomPools": { } }

[medranSolus]

Hello and sorry for long time in responses.
I've reproduced your application allocations and de-allocations in same order but seems I've did not hit any library assert, nor achieved same situation you posted in JSON dump (thanks for that, it helped us narrow what could be the problem here). My suggestion here would be to enable VMA_HEAVY_ASSERT() (by just simply defining it to VMA_ASSERT() for ex.) to perform additional checks inside library that can catch more complex scenarios.
Also are you using VMA in multithreaded application? If so then what method of synchronization are you using, your own or the one from library itself (see VMA_ALLOCATOR_CREATE_EXTERNALLY_SYNCHRONIZED_BIT)? Situation from JSON dump you posted seems like some race conditions occurred that trashed internal state of library, so if you could check that I'd be greatful.
Marek Machliński, Developer Technology Engineer Intern, AMD

[rumblehhh]

No problem!
VMA_HEAVY_ASSERT didn't trigger during the application run - I checked that VmaIsPow2 was called inside VmaAlignUp so VMA_HEAVY_ASSERT was definitely enabled.
The application itself is multithreaded but all vulkan specific calls are made on a single thread so from VMAs perspective its single threaded.
We don't use VMA_ALLOCATOR_CREATE_EXTERNALLY_SYNCHRONIZED_BIT i.e. we let VMA handle its own sync.

I had a play around with disabling some of the additional vma debugging features we use and the issue went away.
In debug builds we currently enable VMA_DEBUG_INITIALIZE_ALLOCATIONS, VMA_DEBUG_DETECT_CORRUPTION and set VMA_DEBUG_MARGIN=4.
If I disable all of these the problem goes away.
If I enable only VMA_DEBUG_MARGIN=4 the problem comes back so it seems related to its usage.
Could VMA be trashing its own state when using this flag?

[medranSolus]

Hi and thanks for checking that!
I've tested said scenario with same settings but sadly could not reproduce the same bug you caught. Maybe you are using some other options that interfere with debug margin that we are not aware of? For now the best workaround I could suggest for you is to disable VMA_DEBUG_MARGIN till we could sort this problem out.
Marek Machliński, Developer Technology Engineer Intern, AMD

[rumblehhh]

No problem.

Maybe you are using some other options that interfere with debug margin that we are not aware of

We're not explicitly enabling any other features but we will set VmaAllocatorCreateInfo::flags based on the implementations supported features.

If it helps I'm using Linux + clang-14.
Are there any additional logs I can gather which might help understand when/why m_NullBlock->offset (this seems to be what determines whether the assert in question is thrown) is being modified incorrectly?

For now the best workaround I could suggest for you is to disable VMA_DEBUG_MARGIN till we could sort this problem out.

If we can figure out the root cause that'd be preferable as using VMA_DEBUG_MARGIN + VMA_DEBUG_INITIALIZE_ALLOCATIONS/VMA_DEBUG_DETECT_CORRUPTION have helped us debug other issues previously.

[medranSolus]

Hi, the problem that is appearing here is caused by improper merging of free blocks during their deletion in your case. When creating new allocation with debug margin there is block placed behind the actual allocation indicating requested margin.

This seems to be causing the whole merging problem. I could dig deeper to that if you could share with me 3 more logs (same as previous JSON dump):

• first one before you start freeing up memory,
• second one after first allocation is made after some deletion,
• third one when you made second allocation after some more deletions.

Maybe that would indicate what is happening next, as it would show me the layout of memory before corruption.
Marek Machliński, Developer Technology Engineer Intern, AMD

[rumblehhh]

We have the following pattern:

1. Lots of allocations
2. Lots of frees
3. One allocation
4. Lots of frees

I'm attaching:

• a log with the allocations/frees
• a log with a json dump before freeing memory (at the start of 2)
• a log with a json dump after first allocation after deletions (just after 3)
• a log with a json dump after first deletion after allocation after deletions (just after the first deletion in 4)

after_first_allocation_after_deletion.txt
after_first_deletion_after_allocation_after_deletions.txt
before_freeing_memory.txt
vma_log.txt

[medranSolus]

Hi and thanks for that logs, they really helped us!
We've created branch fix-debug-margin with proposed fix for that issue, if you could check it out in your situation, we will merge it into main branch.
Marek Machliński, Developer Technology Engineer Intern, AMD

[rumblehhh]

Thanks for the proposed fix! Unfortunately I still see the same assert :(.

If I additionally enable VMA_HEAVY_ASSERT by setting it to VMA_ASSERT I then hit the following assert
virtual bool VmaBlockMetadata_TLSF::Validate() const: Assertion `0 && "Validation failed: " "calculatedFreeSize == GetSumFreeSize()"' failed.

[medranSolus]

Hi, sorry to hear that.
About new validation assert, I fixed it now on the same branch. Could you send me JSON dumps from before starting frees (after point 1 from your previous post) and before deleting allocator (after point 4)? With previous fix I've deleted some corner case in allocator, sadly not related to your issue, but real one can become more apparent now. Thanks for your help so far!
Marek Machliński, Developer Technology Engineer Intern, AMD

[rumblehhh]

Sorry for the delay - I've been OoO for the last week.
No problem glad I can help.
Please see attached the attached files as requested:
before_freeing_memory_2.txt
before_freeing_allocator.txt