Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

space:truncate() (and sometimes even space:delete()) doesn't work when memtx_memory is reached #3807

Open
dsamirov opened this issue Nov 14, 2018 · 5 comments
Assignees
Labels
bug Something isn't working

Comments

@dsamirov
Copy link

Tarantool version: 1.10.1-370-ge93dc30

OS version: centos 7

Bug description:

Sometimes when memtx_memory limit is reached space:truncate() doesn't work with error
Failed to allocate 21 bytes in slab allocator for memtx_tuple.

@dkuk
Copy link

dkuk commented Nov 16, 2018

Same for me Tarantool 1.9.2-7-g3736c3798
Ubuntu 16.10

Also reproduces in Tarantool 1.10.2-66-g2972cfcee, Ubuntu 18.04.1 LTS

Sometimes this bug affects space:delete() operation. It falls with same error. After instance being restarted delete becomes possible.

@kyukhin kyukhin added the bug Something isn't working label Dec 7, 2018
@kyukhin kyukhin added this to the 2.2.0 milestone Dec 7, 2018
@sharonovd sharonovd changed the title space:truncate() doesn't work when memtx_memory is reached space:truncate() (and sometimes even space:delete() )doesn't work when memtx_memory is reached Dec 12, 2018
@sharonovd sharonovd changed the title space:truncate() (and sometimes even space:delete() )doesn't work when memtx_memory is reached space:truncate() (and sometimes even space:delete()) doesn't work when memtx_memory is reached Dec 12, 2018
@kyukhin kyukhin modified the milestones: 2.2.0, 2.3.0 Apr 1, 2019
PersDep added a commit that referenced this issue Dec 5, 2019
In bps_tree_create_leaf we use matras_alloc in case
bps_tree_garbage_pop didn't work out. However it also might not
succeed. Then we need to return NULL instead of dereferencing NULL
pointer.

Prerequisites: #3807
PersDep added a commit that referenced this issue Dec 5, 2019
memtx_index_extent_reserve was always being called from
memtx_space_replace_all_keys. It could lead to slab allocator failure
while reaching memtx_memory limit, which is especially distressing on
space:truncate() and space:delete() invocations. we  Now it is not
being called if not needed, including space:truncate() and
space:delete() cases.

Part of #3807
PersDep added a commit that referenced this issue Dec 5, 2019
Trying to perform space:truncate() while reaching memtx_memory limit we
could experience slab allocator failure on allocation for service tuple
being upserted into service space to keep record of truncations. This
behavior seems to be quite surprising for users. Now we are using
calloc() to allocate tuples in BOX_TRUNCATE_ID space.

Closes: #3807
PersDep added a commit that referenced this issue Dec 6, 2019
Trying to perform space:truncate() while reaching memtx_memory limit we
could experience slab allocator failure on allocation for service tuple
being upserted into service space to keep record of truncations. This
behavior seems to be quite surprising for users. Now we are using
calloc() to allocate tuples in BOX_TRUNCATE_ID space.

Closes: #3807
@kyukhin kyukhin modified the milestones: 2.3.1, 2.3.2 Dec 23, 2019
PersDep added a commit that referenced this issue Dec 25, 2019
In bps_tree_create_leaf we use matras_alloc in case
bps_tree_garbage_pop didn't work out. However it also might not
succeed. Then we need to return NULL instead of dereferencing NULL
pointer.

Part of: #3807
PersDep added a commit that referenced this issue Dec 31, 2019
In bps_tree_create_leaf we use matras_alloc in case
bps_tree_garbage_pop didn't work out. However it also might not
succeed. Then we need to return NULL instead of dereferencing NULL
pointer.

Part of: #3807
PersDep added a commit that referenced this issue Jan 10, 2020
Trying to perform space:truncate() and space:delete() while reaching
memtx_memory limit we could experience slab allocator failure. This
behavior seems to be quite surprising for users. Now we are increasing
memtx quota if needed for truncation or deletion. After performing it
quota is being set back to the previous value if possible, while it
should be so for almost any case, since we are meant to free some space
during deletion or truncation.

Closes #3807
PersDep added a commit that referenced this issue Jan 13, 2020
Trying to perform space:truncate() and space:delete() while reaching
memtx_memory limit we could experience slab allocator failure. This
behavior seems to be quite surprising for users. Now we are increasing
memtx quota if needed for truncation or deletion. After performing it
quota is being set back to the previous value if possible, while it
should be so for almost any case, since we are meant to free some space
during deletion or truncation.

Closes #3807
PersDep added a commit that referenced this issue Jan 13, 2020
Trying to perform space:truncate() and space:delete() while reaching
memtx_memory limit we could experience slab allocator failure. This
behavior seems to be quite surprising for users. Now we are increasing
memtx quota if needed for truncation or deletion. After performing it
quota is being set back to the previous value if possible, while it
should be so for almost any case, since we are meant to free some space
during deletion or truncation.

Closes #3807
PersDep added a commit that referenced this issue Jan 13, 2020
In bps_tree_create_leaf we use matras_alloc in case
bps_tree_garbage_pop didn't work out. However it also might not
succeed. Then we need to return NULL instead of dereferencing NULL
pointer.

Part of: #3807
PersDep added a commit that referenced this issue Jan 13, 2020
Trying to perform space:truncate() and space:delete() while reaching
memtx_memory limit we could experience slab allocator failure. This
behavior seems to be quite surprising for users. Now we are increasing
memtx quota if needed for truncation or deletion. After performing it
quota is being set back to the previous value if possible, while it
should be so for almost any case, since we are meant to free some space
during deletion or truncation.

Closes #3807
PersDep added a commit to tarantool/small that referenced this issue Jan 20, 2020
By default the quota is strict. If it is set to false, quota_use will
allow to overuse the total available memory limit.

Part of tarantool/tarantool#3807
PersDep added a commit that referenced this issue Jan 20, 2020
In bps_tree_create_leaf we use matras_alloc in case
bps_tree_garbage_pop didn't work out. However it also might not
succeed. Then we need to return NULL instead of dereferencing NULL
pointer.

Part of #3807
PersDep added a commit that referenced this issue Jan 20, 2020
Trying to perform space:truncate() and space:delete() while reaching
memtx_memory limit we could experience slab allocator failure. This
behavior seems to be quite surprising for users. Now we are allowing
to overuse memtx quota if needed for truncation or deletion, using flag
in struct quota. After performing truncation or deletion we reset the
flag so that quota can be overused any more.

Closes #3807
PersDep added a commit to tarantool/small that referenced this issue Jan 22, 2020
By default the quota is enabled. If it is set to false, quota_use will
allow to overuse the total available memory limit.

Part of tarantool/tarantool#3807
PersDep added a commit to tarantool/small that referenced this issue Feb 14, 2020
By default the quota is enabled. If it is set to false, quota_use will
allow to overuse the total available memory limit.

Part of tarantool/tarantool#3807
PersDep added a commit that referenced this issue Feb 14, 2020
In bps_tree_create_leaf we use matras_alloc in case
bps_tree_garbage_pop didn't work out. However it also might not
succeed. Then we need to return NULL instead of dereferencing NULL
pointer.

Part of #3807
PersDep added a commit that referenced this issue Feb 14, 2020
Trying to perform space:truncate() and space:delete() while reaching
memtx_memory limit we could experience slab allocator failure. This
behavior seems to be quite surprising for users. Now we are allowing
to overuse memtx quota if needed for truncation or deletion, using flag
in struct quota. After performing truncation or deletion we reset the
flag so that quota can't be overused any more.

Closes #3807
PersDep added a commit that referenced this issue Feb 14, 2020
Trying to perform space:truncate() and space:delete() while reaching
memtx_memory limit we could experience slab allocator failure. Now it
is solved through quota overuse tech. This commit introduces
corresponding tests.

Part of #3807
PersDep added a commit to tarantool/small that referenced this issue Dec 11, 2020
By default the quota is enabled. If it is set to false, quota_use will
allow to overuse the total available memory limit.
In case of disabled quota smalloc() allocates (and frees) large slabs
using malloc so that the quota will be able to shrink back after those
slabs are freed. Test introduced.

Part of tarantool/tarantool#3807
PersDep added a commit that referenced this issue Dec 11, 2020
Trying to perform space:truncate() while reaching memtx_memory limit
we could experience slab allocator failure. This behavior seems to be
quite surprising for users. Now we are allowing to overuse memtx quota
for tuples in space _truncate using flag in struct quota.
Truncate tuples are only being allocated with large slabs using malloc
so that the quota can shrink back when they are freed.
 
Closes #3807
PersDep added a commit to tarantool/small that referenced this issue Dec 12, 2020
By default the quota is enabled. If it is set to false, quota_use will
allow to overuse the total available memory limit.
In case of disabled quota smalloc() allocates (and frees) large slabs
using malloc so that the quota will be able to shrink back after those
slabs are freed. Test introduced.

Part of tarantool/tarantool#3807
PersDep added a commit that referenced this issue Dec 12, 2020
Trying to perform space:truncate() while reaching memtx_memory limit
we could experience slab allocator failure. This behavior seems to be
quite surprising for users. Now we are allowing to overuse memtx quota
for tuples in space _truncate using flag in struct quota.
Truncate tuples are only being allocated with large slabs using malloc
so that the quota can shrink back when they are freed.
 
Closes #3807
PersDep added a commit to tarantool/small that referenced this issue Dec 19, 2020
By default the quota is enabled. If it is set to false, quota_use will
allow to overuse the total available memory limit.
In case of disabled quota smalloc() allocates (and frees) large slabs
using malloc so that the quota will be able to shrink back after those
slabs are freed. Test introduced.

Part of tarantool/tarantool#3807
PersDep added a commit that referenced this issue Dec 19, 2020
Trying to perform space:truncate() while reaching memtx_memory limit
we could experience slab allocator failure. This behavior seems to be
quite surprising for users. Now we are allowing to overuse memtx quota
for tuples in space _truncate using flag in struct quota.
Truncate tuples are only being allocated with large slabs using malloc
so that the quota can shrink back when they are freed.
 
Closes #3807
PersDep added a commit to tarantool/small that referenced this issue Dec 29, 2020
By default the quota is enabled. If it is set to false, quota_use will
allow to overuse the total available memory limit.
In case of disabled quota smalloc() allocates (and frees) large slabs
using malloc so that the quota will be able to shrink back after those
slabs are freed. Test introduced.

Part of tarantool/tarantool#3807
PersDep added a commit that referenced this issue Jan 19, 2021
Trying to perform space:truncate() while reaching memtx_memory limit
we could experience slab allocator failure. This behavior seems to be
quite surprising for users. Now we are allowing to overuse memtx quota
for tuples in space _truncate using flag in struct quota.
Truncate tuples are only being allocated with large slabs using malloc
so that the quota can shrink back when they are freed.
 
Closes #3807
PersDep added a commit that referenced this issue Jan 19, 2021
Trying to perform space:truncate() while reaching memtx_memory limit
we could experience slab allocator failure. This behavior seems to be
quite surprising for users. Now we are allowing to overuse memtx quota
for tuples in space _truncate using flag in struct quota.
Truncate tuples are only being allocated with large slabs using malloc
so that the quota can shrink back when they are freed.
 
Closes #3807
@kyukhin kyukhin removed this from the 1.10.9 milestone Feb 12, 2021
@kyukhin kyukhin removed the tmp label Feb 12, 2021
@kyukhin kyukhin added this to the wishlist milestone Jul 13, 2021
@no1seman
Copy link

Reproducer for truncate:
Tarantool 2.8.3-0-g01023dbc2
Target: Linux-x86_64-RelWithDebInfo

local yaml = require('yaml')

box.cfg({memtx_memory = 32*1024*1024})

local space = box.schema.space.create('test_space', { if_not_exists = true })
space:format({{ name = 'key', type = 'unsigned'}, { name = 'value', type = 'string'}})
space:create_index('primary', { parts = {{ field = 1, type = 'unsigned' }, { field = 2, type = 'string' }}, if_not_exists = true })
space:create_index('secondary', { parts = {{ field = 2, type = 'string' }}, if_not_exists = true })

local count = space:len()

while true do
    count = count + 1
    if not select(1, pcall(function() box.space.test_space:insert({count, 'Key number: '..tostring(count)}) end)) then
        break
    end
    if count % 10000 == 0 then
        print(count)
    end
end

print('"test_space" length: '.. tostring(box.space.test_space:len()))
print('Arena:\n'..yaml.encode(box.slab.info()))

print('Try to truncate space:')
box.space.test_space:truncate()

Log:
Try to truncate space:
2022-03-26 09:58:25.811 [2468] main memtx_engine.c:1320 !> SystemError Failed to allocate 19 bytes in slab allocator for memtx_tuple: Cannot allocate memory

@RunsFor
Copy link
Contributor

RunsFor commented Jun 21, 2022

Same issue at TEST stand on pretty huge (40gb) instance (and multiple replicasets). Used workaround was pretty time-consuming:

  • space:pairs():take(100000):each(function(t) space:delete({t[1], t[2]}) end)
  • box.snapshot() -- do this on replicas too
  • restart replicasets and wait for it to become available
  • space:truncate()

@kyukhin kyukhin removed the teamC label Jun 27, 2022
@kyukhin kyukhin removed this from the wishlist milestone Jun 27, 2022
nshy added a commit to nshy/tarantool that referenced this issue Oct 25, 2023
We hit tarantool#3807 in release/2.11 for release ASAN build with ASAN-friendly
small allocators.

Follow-up tarantool#7327

NO_CHANGELOG=internal
NO_DOC=internal
nshy added a commit to nshy/tarantool that referenced this issue Oct 25, 2023
We hit tarantool#3807 in release/2.11 for release ASAN build with ASAN-friendly
small allocators.

Follow-up tarantool#7327

NO_CHANGELOG=internal
NO_DOC=internal
nshy added a commit to nshy/tarantool that referenced this issue Oct 25, 2023
We hit tarantool#3807 in release/2.11 for release ASAN build with ASAN-friendly
small allocators.

Follow-up tarantool#7327

NO_CHANGELOG=internal
NO_DOC=internal
locker pushed a commit that referenced this issue Oct 26, 2023
We hit #3807 in release/2.11 for release ASAN build with ASAN-friendly
small allocators.

Follow-up #7327

NO_CHANGELOG=internal
NO_DOC=internal
locker pushed a commit that referenced this issue Oct 26, 2023
We hit #3807 in release/2.11 for release ASAN build with ASAN-friendly
small allocators.

Follow-up #7327

NO_CHANGELOG=internal
NO_DOC=internal

(cherry picked from commit 3fbd7fc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

10 participants