New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kill lsa zones #2480

Closed
avikivity opened this Issue Jun 13, 2017 · 13 comments

Comments

Projects
None yet
6 participants
@avikivity
Copy link
Contributor

avikivity commented Jun 13, 2017

LSA uses zones to track contiguous chunks of segments. But zones are artificial objects, and managing them adds latency since they have to be added and removed as a unit.

Here's an idea from an email:

Again I think we should just kill zones. We can have a bitmap a but set for each segment that lsa took over from stdalloc, and another for lsa segments that are free.

We should concentrate freeing from the lowest-numbered segments and allocation from the highest. This will segregate memory into two arenas: stdalloc and lsa. This will require some support from stdalloc - freeing a portion of allocated memory that is not all of the memory or something at the end, but in the beginning -- like munmap() can free a subset of a mapped area. We may also want better integration with the std allocator so we can opportunistically allocate segments, rather than having an aligned_alloc loop.

(well, we can avoid the need for much of the integration by using aligned_alloc(256k) instead of allocating zones, maybe that by itself is already worthwhile).

@tgrabiec

This comment has been minimized.

Copy link
Contributor

tgrabiec commented Jun 13, 2017

About using aligned_alloc, we used to do this, but switched in the following commit:

commit 5b5c003
Author: Tomasz Grabiec tgrabiec@cloudius-systems.com
Date: Fri Aug 7 20:55:06 2015 +0200

lsa: Don't allocate aligned segments

Requiring alignment means that there must be 64K of contiguous space
to allocate each 32K segment. When memory is fragmented, we may fail
to allocate such segment, even though there's plenty of free space.

This especially hurts forward progress of compaction, which frees
segments randomly and relies on the fact that freeing a segment will
make it available to the next segment request.
@avikivity

This comment has been minimized.

Copy link
Contributor

avikivity commented Jun 13, 2017

Perhaps we should stop pretending, and copy core/memory.cc into scylla. Then lsa+std could be tightly integrated. The scylla std allocator would manage std memory in terms of aligned 256k segments, and really large allocations would be satisfied by allocating a number of contiguous segments. A segment could have multiple roles:

  1. broken up for small_pool (maybe small_pool would allocate a complete segment?)
  2. used for lsa
  3. part of a large std allocation

We might drop the segment size to 128k to thread stacks and lsa segments are the same size.

Even before that, I think aligned_alloc would perform better if lsa->std freeing always happened at the lowest-numbered segments.

@gleb-cloudius

This comment has been minimized.

Copy link
Contributor

gleb-cloudius commented Jun 13, 2017

@avikivity

This comment has been minimized.

Copy link
Contributor

avikivity commented Jun 13, 2017

The change I proposed makes std allocation second class relative to lsa, so it would hurt applications that don't want lsa.

@nyh

This comment has been minimized.

Copy link
Contributor

nyh commented Jun 13, 2017

@nyh

This comment has been minimized.

Copy link
Contributor

nyh commented Jun 13, 2017

@avikivity

This comment has been minimized.

Copy link
Contributor

avikivity commented Jun 13, 2017

The changes I proposed assume that lsa will be used for most allocations. Large allocations outside LSA will be more wasteful, and perhaps small allocations sometimes (if the "maybe" is engaged).

A seastar-based file system or block device would not need lsa; neither would a database that caches btree nodes (Scylla is special in that it caches the result of merging several database files). LSA is useful when most objects' sizes have an unknown distribution.

@tgrabiec

This comment has been minimized.

Copy link
Contributor

tgrabiec commented Jun 13, 2017

@avikivity

This comment has been minimized.

Copy link
Contributor

avikivity commented Jun 13, 2017

Would that be more or less equivalent to configuring the seastar allocator
with page_size of 256k instead of 4k?

It would (if we engage the "maybe").

@avikivity

This comment has been minimized.

Copy link
Contributor

avikivity commented Jun 13, 2017

We might make it a compile-time option.

@avikivity

This comment has been minimized.

Copy link
Contributor

avikivity commented Jun 13, 2017

We would also want lsa and std to allocate from opposite ends, to reduce fragmentation; so std would become a first-fit allocator.

@slivne slivne added this to the 2.x milestone Jul 3, 2017

@glommer

This comment has been minimized.

Copy link
Contributor

glommer commented Apr 28, 2018

@avikivity isn't this done? please close if so.

@avikivity

This comment has been minimized.

Copy link
Contributor

avikivity commented Apr 29, 2018

Fixed by 252c5df.

@avikivity avikivity closed this Apr 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment