Massive allocations in unmanaged memory on Dictionary creation

[Takoooooo]

Description

I wanted to use https://github.com/thecoderok/Unidecode.NET this project in my code, but I have realized it allocates a lot of memory. After some investigations, I have understood what the reason for all those allocations was this file
https://github.com/thecoderok/Unidecode.NET/blob/master/src/Unidecoder.Characters.cs
At peak, Dictionary initialization would consume ~65MB of memory(.net5,x64, Release), and ~99% of this memory is unmanaged, which is quite strange for me. For example "Hello World" console app(.net5,x64, Release) on my machine would consume ~8 MB of memory.
Steps to reproduce? Just initialize the dictionary with the data from https://github.com/thecoderok/Unidecode.NET/blob/master/src/Unidecoder.Characters.cs

Configuration

.net5,x64,Win10

Data

(With dictionary initialization)

Just "hello world" app to compare.

Analysis

From dotMemory, I can see what when the app just starts it starts to allocate memory to the heap(prb initializing dictionary) and also starts to massively allocate unmanaged memory. When the allocation to the heap ends some unmanaged memory is also being cleared and stabilizes on 30MB.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[huoyaoyuan]

It seems that diagnostics tools show string content as unmanaged memory.
However, the type has ~1.5K strings. Certainly a string shouldn't consume 30KB memory.

[GrabYourPitchforks]

I temporarily pathed this to the VM because those folks probably have the best awareness of what might be reporting this. Or at least can route it appropriately. :)

[jkotas]

The dictionary initialization method in the library is huge. The JITed code for it is about 1.5MB.

The unmanaged memory allocation that you are seeing is coming from the JIT. The JIT needs the unmanaged memory to create intermediate representation of the method. It is expected that the intermediate representation for 1.5MB method is going to take 10s MB.

Large auto-generated collection initializers are known source of bad performance and crashes. See for example: #8980.

You should open an issue against the library instead. The library should be fixed to use data-driven approach for initialization of the Dictionary.

[jkotas]

Looks like there is an issue on this already: thecoderok/Unidecode.NET#14

[Takoooooo]

The dictionary initialization method in the library is huge. The JITed code for it is about 1.5MB.

The unmanaged memory allocation that you are seeing is coming from the JIT. The JIT needs the unmanaged memory to create intermediate representation of the method. It is expected that the intermediate representation for 1.5MB method is going to take 10s MB.

Large auto-generated collection initializers are known source of bad performance and crashes. See for example: #8980.

You should open an issue against the library instead. The library should be fixed to use data-driven approach for initialization of the Dictionary.

I`m sorry, but what do you mean by the "data-driven approach for initialization of the Dictionary"?

[huoyaoyuan]

Store the corresponding data in embedded binary file, primitive array of constants, or ReadOnlySpan<byte> that backed by a constant array. Then use a loop to convert the data into dictionary.

[Takoooooo]

Store the corresponding data in embedded binary file, primitive array of constants, or ReadOnlySpan<byte> that backed by a constant array. Then use a loop to convert the data into dictionary.

I tried making one array of integers and a jagged array of strings and to add them in for loop to the dictionary, but it doesn't really solve the issue.Still~65MB

[huoyaoyuan]

a jagged array of strings

This is still codeful. String is not considered primitive in this case.

You can inspect the output assembly with ILSpy, and examine the IL size of method body.