-
Notifications
You must be signed in to change notification settings - Fork 190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Strange exception from unordered_set with memory_stack, on MSVC #144
Comments
The issue with that approach is that it can waste memory. For example, before it allocates 4096 bytes then 8192, then 16384 - all of which are multiple of the page size and can thus be served by allocating whole pages using a page allocator. If we then add the offset on top we need to start an entire new page to store the extra pointer. I think the correct fix here is to allocate the extra capacity in your stack up front. |
What does that look like, exactly? You mean in client code, have |
That should work, yes. You can use |
Thanks; that does seem to fix the issue for the case I posted. However, when trying to apply the fix locally, I noticed another related case, using a node pool allocator:
Here, it gets the same Does this represent a bug inside |
They do, the second parameter is the number of bytes, not number of nodes. You can likewise use |
Ah, sorry for the confusion. I am still seeing a crash though (see end). I was thrown off by the fact that So I guess you mean a user should do something like this:
That does indeed work, for me. However, if a user wanted to grow by (approximately) some number of bytes rather than nodes, it seems they would need to do their own math, perhaps like:
That case still crashes, for me. The math inside So, working backwards from that, here is another crash, which doesn't try to do any weird math in client code:
The number 42 is chosen there to make the math work out to an even 1024 again, so it is the same crash, just achieved in a different way. Previously, with the number 1000, it was working by accident, it seems. Or is there some other |
I don't think there is anything I can do on the library side about it. MSVC wants to allocate a big power of two, which the allocator isn't designed to handle -- it also doesn't need to, allocating big memory pages is perfectly fine for the default allocator. The fundamental issue is that unordered_map is a hybrid container: it allocates both arrays and individual nodes, and the pool should only be used for the latter. |
Ok, fair enough; thank you for sticking with me through the explanation (: So it seems like a node pool allocator is just a bad fit for the use-case. I was just thrown off by the fact that MSVC fails where others did not. But I guess in theory gcc and others would have the same problem, given the right circumstances... |
With the Microsoft compiler (I'm using VC19, but I imagine it's the same for others), and with fairly simple usage of
std::unordered_set
with amemory_stack
allocator, I get an exception coming from inside the STL.I believe it is related to the extra
implementation_offset
reserved by foonathan.I'm not sure if this represents a bug in the MSVC STL, a bug in foonathan::memory, or if this is some expected limitation.
Below is a simple failing test.
The
insert()
line will throwstd::length_error
with messageinvalid hash bucket count
.This is the line in the STL that generates the error: https://github.com/microsoft/STL/blob/2f8342a3a57fb157d881c6a2d42a917d20e413f8/stl/inc/xhash#L1717
Feel free to step through it yourself, but my brief summary:
max_size
comes back as slightly smaller than a power of 2, because ofimplementation_offset
_Floor_of_log_2(...)
)In particular, there is this comment:
I am not sure if this invariant is somehow guaranteed by the C++ standard, or is just an assumption made in the MSVC STL.
It all works fine on glibc, for what that's worth.
I wonder if the
implementation_offset
amount should be something in addition to the basic grow-by-doubling calculation, rather than eating into it, slightly?The text was updated successfully, but these errors were encountered: