mimalloc >= 3.3.0 causes segmentation faults when used from multiple threads

[adamreeve]

We have a .NET library, ParquetSharp, which wraps the Apache Arrow/Parquet C++ library. Apache Arrow uses mimalloc as its default allocator.

When trying to upgrade mimalloc recently, I found that our benchmark tests would consistently crash with a segmentation fault. Versions 3.2.7 and 3.2.8 worked fine, but versions 3.3.0, 3.3.1 and 3.3.2 all eventually crashed with the same error:

Details

Thread 15 "SegfaultRepro" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fdfb6ffe6c0 (LWP 238809)]
0x00007fdfa9b2b353 in _mi_theap_default_set (theap=0x7fdfabd3e700 <theap_main>) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/init.c:938
938       mi_assert_internal(theap->tld->thread_id==0 || theap->tld->thread_id==_mi_thread_id());
(gdb) bt
#0  0x00007fdfa9b2b353 in _mi_theap_default_set (theap=0x7fdfabd3e700 <theap_main>) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/init.c:938
#1  0x00007fdfa9b2ae65 in _mi_thread_init_theap_default () at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/init.c:610
#2  0x00007fdfa9b2b16d in mi_thread_init () at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/init.c:709
#3  0x00007fdfa9b36dab in _mi_malloc_generic (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16455, zero_huge_alignment=0, usable=0x0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/page.c:984
#4  0x00007fdfa9b10189 in mi_theap_malloc_generic (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16447, zero=false, huge_alignment=0, usable=0x0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc.c:176
#5  _mi_theap_malloc_zero_ex (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16447, zero=false, huge_alignment=0, usable=0x0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc.c:239
#6  _mi_theap_malloc_zero (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16447, zero=false, usable=0x0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc.c:244
#7  0x00007fdfa9b16d4e in mi_theap_malloc_zero_no_guarded (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16447, zero=false, usable=0x0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc-aligned.c:57
#8  0x00007fdfa9b16f67 in mi_theap_malloc_zero_aligned_at_overalloc (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16384, alignment=64, offset=0, zero=false, usable=0x0)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc-aligned.c:94
#9  0x00007fdfa9b17364 in mi_theap_malloc_zero_aligned_at_generic (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16384, alignment=64, offset=0, zero=false, usable=0x0)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc-aligned.c:179
#10 0x00007fdfa9b1760c in mi_theap_malloc_zero_aligned_at (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16384, alignment=64, offset=0, zero=false, usable=0x0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc-aligned.c:230
#11 0x00007fdfa9b1764b in mi_theap_malloc_aligned_at (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16384, alignment=64, offset=0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc-aligned.c:239
#12 0x00007fdfa9b1767a in mi_theap_malloc_aligned (theap=0x7fdfabbd7140 <_mi_theap_empty>, size=16384, alignment=64) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc-aligned.c:243
#13 0x00007fdfa9b177d7 in mi_malloc_aligned (size=16384, alignment=64) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/v3.3.1-c528aac713.clean/src/alloc-aligned.c:274
#14 0x00007fdfa8efa5e4 in arrow::(anonymous namespace)::MimallocAllocator::AllocateAligned (size=16384, alignment=64, out=0x7fdfb6ffc950) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/memory_pool.cc:404
#15 0x00007fdfa8efc9a9 in arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::MimallocAllocator>::Allocate (this=0x7fdfabd3a5c0 <arrow::global_state+576>, size=16384, alignment=64, out=0x7fdfb6ffc950)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/memory_pool.cc:473
#16 0x00007fdfa8f011d8 in arrow::PoolBuffer::Reserve (this=0x7fdfac0f9d70, capacity=16384) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/memory_pool.cc:932
#17 0x00007fdfa8f014d4 in arrow::PoolBuffer::Resize (this=0x7fdfac0f9d70, new_size=16384, shrink_to_fit=true) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/memory_pool.cc:956
#18 0x00007fdfa8efbf9d in arrow::(anonymous namespace)::ResizePoolBuffer<std::unique_ptr<arrow::ResizableBuffer>, std::unique_ptr<arrow::PoolBuffer> > (buffer=..., size=16384)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/memory_pool.cc:1003
#19 0x00007fdfa8efbabb in arrow::AllocateResizableBuffer (size=16384, alignment=64, pool=0x7fdfabd3a5c0 <arrow::global_state+576>) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/memory_pool.cc:1030
#20 0x00007fdfa82eae6d in arrow::BufferBuilder::Resize (this=0x7fdfac0f6d20, new_capacity=16384, shrink_to_fit=true) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/buffer_builder.h:78
#21 0x00007fdfa8440df5 in arrow::TypedBufferBuilder<arrow::internal::HashTable<arrow::internal::ScalarMemoTable<int, arrow::internal::HashTable>::Payload>::Entry, void>::Resize (this=0x7fdfac0f6d20, new_capacity=1024, shrink_to_fit=true)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/buffer_builder.h:287
#22 0x00007fdfa8440375 in arrow::internal::HashTable<arrow::internal::ScalarMemoTable<int, arrow::internal::HashTable>::Payload>::UpsizeBuffer (this=0x7fdfac0f6d00, capacity=1024)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/util/hashing.h:351
#23 0x00007fdfa843e914 in arrow::internal::HashTable<arrow::internal::ScalarMemoTable<int, arrow::internal::HashTable>::Payload>::HashTable (this=0x7fdfac0f6d00, pool=0x7fdfabd3a5c0 <arrow::global_state+576>, capacity=1024)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/util/hashing.h:252
#24 0x00007fdfa843cc4e in arrow::internal::ScalarMemoTable<int, arrow::internal::HashTable>::ScalarMemoTable (this=0x7fdfac0f6cf8, pool=0x7fdfabd3a5c0 <arrow::global_state+576>, entries=1024)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/arrow/util/hashing.h:423
#25 0x00007fdfa84137af in parquet::(anonymous namespace)::DictEncoderImpl<parquet::PhysicalType<(parquet::Type::type)1> >::DictEncoderImpl (this=0x7fdfac0f6ca0, desc=0x7fdfac0f6c40, pool=0x7fdfabd3a5c0 <arrow::global_state+576>, __in_chrg=<optimized out>,
    __vtt_parm=<optimized out>) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/parquet/encoder.cc:472
#26 0x00007fdfa840f820 in std::make_unique<parquet::(anonymous namespace)::DictEncoderImpl<parquet::PhysicalType<(parquet::Type::type)1> >, parquet::ColumnDescriptor const*&, arrow::MemoryPool*&> () at /usr/include/c++/15/bits/unique_ptr.h:1084
#27 0x00007fdfa8408a26 in parquet::MakeEncoder (type_num=parquet::Type::INT32, encoding=parquet::Encoding::RLE_DICTIONARY, use_dictionary=true, descr=0x7fdfac0f6c40, pool=0x7fdfabd3a5c0 <arrow::global_state+576>)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/parquet/encoder.cc:1773
#28 0x00007fdfa8363c87 in parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)1> >::TypedColumnWriterImpl (this=0x7fdfac0148b0, metadata=0x7fdfac0dde90, pager=std::unique_ptr<parquet::PageWriter> = {...}, use_dictionary=true,
    encoding=parquet::Encoding::RLE_DICTIONARY, properties=0x7fdfac013ec0, bloom_filter=0x0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/parquet/column_writer.cc:1291
#29 0x00007fdfa8361859 in std::_Construct<parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)1> >, parquet::ColumnChunkMetaDataBuilder*&, std::unique_ptr<parquet::PageWriter, std::default_delete<parquet::PageWriter> >, bool const&, parquet::Encoding::type&, parquet::WriterProperties const*&, parquet::BloomFilter*&> (__p=0x7fdfac0148b0) at /usr/include/c++/15/bits/stl_construct.h:133
#30 0x00007fdfa835e68c in std::allocator_traits<std::allocator<void> >::construct<parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)1> >, parquet::ColumnChunkMetaDataBuilder*&, std::unique_ptr<parquet::PageWriter, std::default_delete<parquet::PageWriter> >, bool const&, parquet::Encoding::type&, parquet::WriterProperties const*&, parquet::BloomFilter*&> (__p=0x7fdfac0148b0) at /usr/include/c++/15/bits/alloc_traits.h:805
#31 std::_Sp_counted_ptr_inplace<parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)1> >, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<parquet::ColumnChunkMetaDataBuilder*&, std::unique_ptr<parquet::PageWriter, std::default_delete<parquet::PageWriter> >, bool const&, parquet::Encoding::type&, parquet::WriterProperties const*&, parquet::BloomFilter*&> (this=0x7fdfac0148a0, __a=...) at /usr/include/c++/15/bits/shared_ptr_base.h:606
#32 0x00007fdfa835ac39 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)1> >, std::allocator<void>, parquet::ColumnChunkMetaDataBuilder*&, std::unique_ptr<parquet::PageWriter, std::default_delete<parquet::PageWriter> >, bool const&, parquet::Encoding::type&, parquet::WriterProperties const*&, parquet::BloomFilter*&> (this=0x7fdfb6ffd5a8, __p=@0x7fdfb6ffd5a0: 0x0, __a=...) at /usr/include/c++/15/bits/shared_ptr_base.h:969
#33 0x00007fdfa83513be in std::__shared_ptr<parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)1> >, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<void>, parquet::ColumnChunkMetaDataBuilder*&, std::unique_ptr<parquet::PageWriter, std::default_delete<parquet::PageWriter> >, bool const&, parquet::Encoding::type&, parquet::WriterProperties const*&, parquet::BloomFilter*&> (this=0x7fdfb6ffd5a0, __tag=...) at /usr/include/c++/15/bits/shared_ptr_base.h:1719
#34 0x00007fdfa8346b24 in std::shared_ptr<parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)1> > >::shared_ptr<std::allocator<void>, parquet::ColumnChunkMetaDataBuilder*&, std::unique_ptr<parquet::PageWriter, std::default_delete<parquet::PageWriter> >, bool const&, parquet::Encoding::type&, parquet::WriterProperties const*&, parquet::BloomFilter*&> (this=0x7fdfb6ffd5a0, __tag=...) at /usr/include/c++/15/bits/shared_ptr.h:463
#35 0x00007fdfa833eff6 in std::make_shared<parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)1> >, parquet::ColumnChunkMetaDataBuilder*&, std::unique_ptr<parquet::PageWriter, std::default_delete<parquet::PageWriter> >, bool const&, parquet::Encoding::type&, parquet::WriterProperties const*&, parquet::BloomFilter*&> () at /usr/include/c++/15/bits/shared_ptr.h:1008
#36 0x00007fdfa8326873 in parquet::ColumnWriter::Make (metadata=0x7fdfac0dde90, pager=std::unique_ptr<parquet::PageWriter> = {...}, properties=0x7fdfac013ec0, bloom_filter=0x0)
    at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/parquet/column_writer.cc:2722
#37 0x00007fdfa8469db1 in parquet::RowGroupSerializer::CreateColumnWriterForColumn (this=0x7fdfac0104d0, col_meta=0x7fdfac0dde90, column_ordinal=0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/parquet/file_writer.cc:324
#38 0x00007fdfa8468de7 in parquet::RowGroupSerializer::NextColumn (this=0x7fdfac0104d0) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/parquet/file_writer.cc:150
#39 0x00007fdfa8467237 in parquet::RowGroupWriter::NextColumn (this=0x7fdfac010580) at /home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/arrow/src/e-arrow-24-991446d5c5.clean/cpp/src/parquet/file_writer.cc:56
#40 0x00007fdfa818b67a in RowGroupWriter_NextColumn (row_group_writer=0x7fdfac010580, column_writer=0x7fdfb6ffda38) at /home/adam/dev/gross/ParquetSharp/cpp/RowGroupWriter.cpp:28
#41 0x00007fff7918f8c3 in ?? ()
#42 0x00007fdfb6ffda38 in ?? ()
#43 0x00007fdfb6ffda38 in ?? ()
#44 0x0000000000000001 in ?? ()

This is from x86_64 Fedora Linux 43 with gcc 15.2.1, and also reproduced on an Ubuntu 24.04 GitHub actions runner.

There's some more context at G-Research/ParquetSharp#647.

If I change the code to not use multiple threads, the problem goes away.

I tried to reproduce this with a minimal C++ program that only uses mimalloc, but haven't had any success yet. But Apache Arrow only uses mi_malloc_aligned, mi_realloc_aligned and mi_free in normal usage, so it shouldn't be doing anything too complicated (see https://github.com/apache/arrow/blob/060062178ca85fa2d7dbd4083574bca6f91cc44c/cpp/src/arrow/memory_pool.cc#L396).

Are there any more details I can provide or experiments I can try that would help determine what the problem is? I tried using MIMALLOC_SHOW_ERRORS=1 and MIMALLOC_VERBOSE=1 but that didn't seem to show anything interesting:

Details

Run 0
mimalloc: process init: 0x7FD11DFFF6C0
v3.3.1, debug (built on Apr 29 2026, 15:28:39)
option 'show_errors': 1
option 'show_stats': 1
option 'verbose': 1
option 'deprecated_eager_commit': 1
option 'arena_eager_commit': 2
option 'purge_decommits': 1
option 'allow_large_os_pages': 0
option 'reserve_huge_os_pages': 0
option 'reserve_huge_os_pages_at': -1
option 'reserve_os_memory': 0 KiB
option 'deprecated_segment_cache': 0
option 'deprecated_page_reset': 0
option 'deprecated_abandoned_page_purge': 0
option 'deprecated_segment_reset': 0
option 'deprecated_eager_commit_delay': 1
option 'purge_delay': 1000
option 'use_numa_nodes': 0
option 'disallow_os_alloc': 0
option 'os_tag': 100
option 'max_errors': 32
option 'max_warnings': 32
option 'deprecated_max_segment_reclaim': 10
option 'destroy_on_exit': 0
option 'arena_reserve': 1048576 KiB
option 'arena_purge_mult': 1
option 'deprecated_purge_extend_delay': 1
option 'disallow_arena_alloc': 0
option 'retry_on_oom': 400
option 'visit_abandoned': 0
option 'guarded_min': 0
option 'guarded_max': 1073741824
option 'guarded_precise': 0
option 'guarded_sample_rate': 0
option 'guarded_sample_seed': 0
option 'generic_collect': 10000
option 'page_reclaim_on_free': 0
option 'page_full_retain': 2
option 'page_max_candidates': 4
option 'max_vabits': 0
option 'pagemap_commit': 0
option 'page_commit_on_demand': 0
option 'page_max_reclaim': -1
option 'page_cross_thread_max_reclaim': 32
option 'allow_thp': 1
option 'minimal_purge_size': 0 KiB
option 'arena_max_object_size': 2097152 KiB
debug level : 3
secure level: 0
mem tracking: none
guarded build: disabled
mimalloc: reserved 1048576 KiB memory
Run 1
Run 2
(exit with code 139)

[daanx]

Hmm, that is not good; I will look into this when I can. I see that the crash is in an assertion -- does it also crash in release mode?

Can you try the latest dev3 and run? I added an extra assertion that checks if theap->tld != NULL (just wondering if that causes the segfault).

Can you also try to run with the environment variable MIMALLOC_ALLOW_THP=0?

[adamreeve]

Yes it also crashes in release mode but I don't have debug symbols enabled there so the backtrace isn't very useful.

I tested the dev3 branch (commit 7e6c5506102be80d087f21b4628f728f4f5a3610) and the new assertion does fail:

mimalloc: assertion failed: at "/home/adam/dev/gross/ParquetSharp/build/vcpkg/buildtrees/mimalloc/src/8f4f5a3610-e0cba1e824.clean/src/init.c":949, _mi_theap_default_set
  assertion: "theap->tld != NULL"

Running with MIMALLOC_ALLOW_THP=0 didn't seem to make any difference, I get the same assertion failure on the dev3 branch.

[adamreeve]

I've reduced my repro down to a .NET program that calls into libmimalloc directly. It seems like this might be related to how the .NET runtime manages threads as I haven't been able to reproduce it with a C++ program using pthread:

using System.Runtime.InteropServices;

namespace SegfaultRepro;

public static class Program
{
    public static void Main()
    {
        for (var i = 0; i < 10; ++i)
        {
            Console.WriteLine($"Run {i}");
            Thread thread = new Thread(RunMimallocThread);
            thread.Start();
            thread.Join();
        }
    }

    private static void RunMimallocThread()
    {
        var ptr = mi_malloc_aligned(16384, 64);
        mi_free(ptr);
    }

    private const string MimallocLocation = "/home/adam/dev/mimalloc-repro/mimalloc/out/debug/libmimalloc-debug.so";

    [DllImport(MimallocLocation)]
    private static extern IntPtr mi_malloc_aligned(ulong size, ulong alignment);

    [DllImport(MimallocLocation)]
    private static extern void mi_free(IntPtr ptr);
}

If you have the .NET SDK installed (I have version 10.0.106), you should be able to put that in a test.cs file, adjust MimallocLocation, and call dotnet run test.cs to reproduce this.

[daanx]

Ah great -- thanks for the quick respons. That theap->tld==NULL is not ok; Thanks for the potential repro! I'll look into it further.

[daanx]

(I am at a workshop now so cannot test yet, but if you replace:

public static void Main()
    {
        for (var i = 0; i < 10; ++i)

by

public static void Main()
    {
        RunMimallocThread()
        for (var i = 0; i < 10; ++i)

does the crash then still happen?

[adamreeve]

No the crash doesn't happen with that change (I tested a few times and also increased the loop count, and tried a release build)

[daanx]

Ah, then I know what the issue is. I'll need to think a bit what the proper fix is. (the issue that the main thread is misidentified and mimalloc cleans up the "main" thread at some point and afterwards things go wrong. When you add the extra call that thread becomes main and everything goes fine as that thread keeps being alive).
Thanks for the quick responses !