Coredumps during restart_then_repair_node nemesis #4918
Comments
|
@ShlomiBalalis is it a regression? new test? |
|
These look like |
|
@tzach I'm not sure if it's regression, since we didn't always use that flag |
|
On Wed, Aug 28, 2019 at 9:28 PM Benny Halevy ***@***.***> wrote:
These look like abort-on-lsa-bad-alloc.
Yes, looks like this option is on.
@tgrabiec <https://github.com/tgrabiec> please confirm.
Also, it looks like the flag might be causing too many false alarms.
Why do you think so?
… Do we still need it by default?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4918>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACFIL4AFF4K7OG2WKWB3PTQG3GUNANCNFSM4IRNGA4A>
.
|
If I understand correctly, those errors could be benign, no? |
|
On Thu, Aug 29, 2019 at 4:07 PM Benny Halevy <notifications@github.com>
wrote:
On Wed, Aug 28, 2019 at 9:28 PM Benny Halevy *@*.***> wrote: These look
like abort-on-lsa-bad-alloc.
Yes, looks like this option is on.
@tgrabiec <https://github.com/tgrabiec> https://github.com/tgrabiec
please confirm. Also, it looks like the flag might be causing too many
false alarms.
Why do you think so?
If I understand correctly, those errors could be benign, no?
If the memory pressure is transient and scylla survives it should we fail
the test?
If it's transient, then we could recover once memory consumption lowers,
but it could be argued that the system should always protect itself so that
a condition in which we OOM is never reached. We don't have control over
which allocation will fail when we're OOM and there are some critical
memory allocations which when fail, we would have no choice than abort the
server. So if we recover, then by pure chance.
… |
|
@denesb They're suppose to be different coredumps |
|
That is this `.gz.aa` format? Maybe I'm trying to extract with the
wrong tool? Firefox identifies them as 'Audible Audiobook' (or
something similar) which suggests they were renamed.
…On Mon, 2019-09-09 at 05:53 -0700, Shlomi Balalis wrote:
@denesb They're suppose to be different coredumps
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
[
{
***@***.***": "http://schema.org",
***@***.***": "EmailMessage",
"potentialAction": {
***@***.***": "ViewAction",
"target": "
#4918
",
"url": "
#4918
",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
***@***.***": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
|
|
@ShlomiBalalis how do we extract one coredump ? |
|
I simply renamed one |
|
Manged to load a core in |
|
Deleted |
|
The cores don't make any sense (tried 3 so far -- all look the same). Also, looking at frame 4: Makes no sense. |
|
Finally in the last core (https://storage.cloud.google.com/upload.scylladb.com/core.scylla.996.9f0393fe20f04dfab829e5bb5cc4bdad.20078.1566947830000000.gz/core.scylla.996.9f0393fe20f04dfab829e5bb5cc4bdad.20078.1566947830000000.gz.aa) I see something more sane. |
|
So far this looks very similar to #4951. We are trying to allocate for a reasonable sized |
|
We are not resharding ... (at least according to the title :) ).
…On Tue, Sep 10, 2019 at 1:04 PM Botond Dénes ***@***.***> wrote:
So far this looks very similar to #4951
<#4951>.
(gdb) scylla memory
Used memory: 7742214144
Free memory: 21442560
Total memory: 7763656704
LSA:
allocated: 1544028160
used: 1541013504
free: 3014656
Cache:
total: 262144
used: 137056
free: 125088
Memtables:
total: 1543766016
Regular:
real dirty: 1540620288
virt dirty: 1451728712
System:
real dirty: 131072
virt dirty: 131072
Streaming:
real dirty: 0
virt dirty: 0
Small pools:
objsz spansz usedobj memory wst%
1 4096 0 0 0.0
1 4096 0 0 0.0
1 4096 0 0 0.0
1 4096 0 0 0.0
2 4096 0 0 0.0
2 4096 0 0 0.0
3 4096 0 0 0.0
3 4096 0 0 0.0
4 4096 0 0 0.0
5 4096 0 0 0.0
6 4096 0 0 0.0
7 4096 0 0 0.0
8 4096 25380 204800 0.9
10 4096 0 8192 99.9
12 4096 1 8192 99.8
14 4096 0 8192 99.8
16 4096 13558 217088 0.1
20 4096 1797 36864 2.1
24 4096 18994 458752 0.2
28 4096 640 20480 12.3
32 4096 14203 458752 0.9
40 4096 7429 299008 0.2
48 4096 1551 81920 8.7
56 4096 15781 892928 0.8
64 4096 26061 1675264 0.4
80 4096 74299 5971968 0.1
96 4096 24946 2441216 0.3
112 4096 58216 6631424 0.1
128 4096 24580 3158016 0.4
160 4096 27022 4440064 0.3
192 4096 13042 2560000 0.6
224 4096 5029 1159168 1.3
256 4096 154 49152 19.8
320 8192 8193 2711552 1.0
384 8192 4300 1703936 1.5
448 4096 404 212992 13.5
512 4096 8004 4124672 0.6
640 12288 315 245760 16.9
768 12288 4836 3735552 0.6
896 8192 93 147456 41.9
1024 4096 8358 8634368 0.9
1280 20480 484 688128 8.2
1536 12288 11 131072 80.9
1792 16384 126 294912 21.9
2048 8192 295 622592 3.0
2560 20480 102 458752 41.3
3072 12288 18 245760 77.5
3584 28672 -1 294912 98.4
4096 16384 244 1146880 12.9
5120 20480 12 425984 81.7
6144 24576 70 491520 12.5
7168 28672 23 622592 70.2
8192 32768 11772 96894976 0.5
10240 40960 5 851968 90.1
12288 49152 13 983040 83.8
14336 57344 74 1245184 11.5
16384 65536 601 10027008 1.8
Small allocations: 167723008 [B]
Page spans:
index size [B] free [B] large [B] [spans]
0 4096 4096 4096 1
1 8192 8192 0 0
2 16384 0 0 0
3 32768 0 621379584 18963
4 65536 65536 133562368 2038
5 131072 131072 3268280320 24935
6 262144 7602176 216268800 825
7 524288 13631488 401080320 765
8 1048576 0 2864709632 2732
9 2097152 0 2097152 1
10 4194304 0 0 0
11 8388608 0 0 0
12 16777216 0 0 0
13 33554432 0 0 0
14 67108864 0 67108864 1
15 134217728 0 0 0
16 268435456 0 0 0
17 536870912 0 0 0
18 1073741824 0 0 0
19 2147483648 0 0 0
20 4294967296 0 0 0
21 8589934592 0 0 0
22 17179869184 0 0 0
23 34359738368 0 0 0
24 68719476736 0 0 0
25 137438953472 0 0 0
26 274877906944 0 0 0
27 549755813888 0 0 0
28 1099511627776 0 0 0
29 2199023255552 0 0 0
30 4398046511104 0 0 0
31 8796093022208 0 0 0
Large allocations: 7574491136 [B]
We are trying to allocate for a reasonable sized frozen_mutation (52K)
and fail while trying to increase our reserves, due to a largish
_non_lsa_reserve:
(gdb) p shard_segment_pool._non_lsa_reserve
$8 = 87'105'536
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#4918>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AA2OCCGDXVHSJSCQB4WGRK3QI5WKXANCNFSM4IRNGA4A>
.
|
|
Again, we have lots of large buffers and one quite large (67MB) as well: There are 25K 128K buffers. |
|
The 67M one is most likely the memory allocator's vector of page descriptors. |
|
There is no resharding now, only 5 regular compactions: |
|
There are over 9K |
|
I looked up all virtual objects from memory and counted them by type: We have a huge number of continuations (101K !!) and loads of readers. Yet we have almost no active tasks executing: I'll try to find the queue where these reads are. |
|
|
Due to the large number of The object shouldn't be of size 24 in the first place as I'm pretty sure that is less than the minimum possible size of an Another example (size sounds more sane): Smp message queue object pointers should be in the range |
|
@avikivity please take a look so we can decide if this is a must for 3.1.0 |
|
@avikivity please take a look so we can decide if this is a must for 3.1.0 |
|
I tried again opening those cores that I failed to earlier, this time paired with the right binary (the on e from the AMI) and I succeed. Looks like we may be able to examine the other cores as well. For now I'll try to take a closer look at #5111 which is the same large partition workload and has a core very similar to some of those we've seen here. That issue also has metrics available. |
|
On Mon, Sep 30, 2019, 21:32 Botond Dénes ***@***.***> wrote:
There is nothing like that in the logs. The node that is supposed to be
the master (node-5) is attempting to booststrap pretty much all along and
fails as the other two nodes (node-1 and node-4 keep crashing).
If there is no repair log on any of the nodes, there is no repair was
performed.
—
… You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4918>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACOETGZJZ2ZMH7LMTFNVGLQMH5YPANCNFSM4IRNGA4A>
.
|
|
Opened #5123 for the querier_cache::stats::population mis-accounting. |
1 task
|
It seems to have occurred again with rc9 (ami-09c8d17ff81a2773e in ireland): Translated: |
@ShlomiBalalis this looks more like #4978 |
|
moving this to 3.2 |
|
if this reproduces on master / 3.2 please post and we will assign and analyze (/ backport) |
@denesb what would it take to restore this metadata in mc format? |
You mean add it back, so we'll have it in the future? Or in the context of this core.... somehow extract it from memory? We can certainly change our code to store it, but we can't store it in the regular metadata component, that would not be I don't know why they removed it in the first place. Was it deemed unuseful? /cc @raphaelsc |
I mean add it back so we have it available again in the future.
Right.
|
|
Hit with Core is here: https://jenkins.scylladb.com/view/scylla-3.3/job/scylla-3.3/job/dtest-release/10/artifact/logs-release.2/1580837245451_read_amplification_test.ReadAmplificationTest.no_read_amplification_on_repair_with_mv_test/node4-scylla.44733.1580837244.core.gz |
|
@bhalevy I know we did some fixes around memeory management - are we still hitting this - if not please close |
|
The dtest is passing reliably so we can close this issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.
Installation details
Scylla version (or git commit hash):3.1.0.rc4-0.20190826.e4a39ed31
Cluster size:4
OS (RHEL/CentOS/Ubuntu/AWS AMI):ami-0ececa5cacea302a8
During restart_then_repair_node, the target node (# 5) suffered from streaming exceptions:
While 2 other nodes suffered from semaphore timeouts (could be related to #4615)
and created coredumps like so:
Here I'll add links to all of those kind of coredumps, knowing that there's currently a bit of an issue with uploading them, hoping that one of them uploaded correctly:
Different backtraces and translation during the run:
translated:
Translation:
The other node's coredumps:
Other download locaions:
relevant journalctl logs of the nodes can be found on
scratch.scylladb.com/shlomib/longevity-large-partitions-4d-db-cluster.tar
The text was updated successfully, but these errors were encountered: