Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix limits #100

Merged
merged 3 commits into from Jan 8, 2019

Conversation

3 participants
@pmconrad
Copy link

commented Dec 30, 2018

Limit allocation during deserialization

@jmjatlanta

This comment has been minimized.

Copy link

commented Dec 31, 2018

Please validate my interpretation of this change below. Am I missing some other benefit? If there is more discussion at STEEM, would you please provide a link?

--
While unpacking, we guess at the number of elements in a packed vector. We can easily overestimate, and reserve too many elements, thereby wasting memory, which will be reclaimed when we resize later in the process.

To reduce over-reserving, we now pick the minimum of (the estimated size or an arbitrary number of 200 ). While this may cause re-allocations, it is better than over-allocating.

@pmconrad

This comment has been minimized.

Copy link
Author

commented Dec 31, 2018

Yes, basically. Related STEEM PRs:

@abitmore

This comment has been minimized.

Copy link
Member

commented Jan 1, 2019

It's mentioned in steemit/steem#3201 that resize() initializes but not only pre-allocates, thus may impact performance. Perhaps we can change them all to reserve()?

@pmconrad

This comment has been minimized.

Copy link
Author

commented Jan 2, 2019

I don't see how to get around the initialization.
One option would be to use reserve and a temporary variable for the nested unpack call, then emplace_back( std::move(tmp) ), similar to

value.reserve( std::min( size.value, FC_MAX_PREALLOC_SIZE ) );
for( uint32_t i = 0; i < size.value; ++i )
{
std::pair<K,V> tmp;
fc::raw::unpack( s, tmp, _max_depth );
value.insert( std::move(tmp) );
}
. Obviously, creating the temporary variable also means one empty initialization for every element to be deserialized, so this is not an improvement.
Better ideas?

@abitmore
Copy link
Member

left a comment

Makes sense.

Show resolved Hide resolved include/fc/config.hpp Outdated
Show resolved Hide resolved include/fc/container/flat.hpp
Show resolved Hide resolved include/fc/container/flat.hpp

@pmconrad pmconrad merged commit b2e738a into master Jan 8, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@pmconrad pmconrad deleted the fix_limits branch Jan 8, 2019

@pmconrad

This comment has been minimized.

Copy link
Author

commented Jan 8, 2019

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.