Error in `/bin/java': corrupted double-linked list: 0x00007fd040187870

[aleksbykov]

During job https://jenkins.scylladb.com/view/scylla-4.2/job/scylla-4.2/job/longevity/job/longevity-50gb-4days-test/13
after 2 hours of running one of cassandra stress command terminated with next error:

** Error in `/bin/java': corrupted double-linked list: 0x00007fd040187870 ***
total,       5052552,     540,     540,     540,    32.9,     8.9,   141.7,   274.5,   454.8,   595.6, 1290.0,  0.01230,      0,      0,       0,       0,       0,       0
======= Backtrace: =========
/lib64/libc.so.6(+0x7f7c4)[0x7fd0a6da47c4]
/lib64/libc.so.6(+0x82f88)[0x7fd0a6da7f88]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7fd0a6daaadc]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x8bbb4d)[0x7fd0a65f0b4d]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x45fdd9)[0x7fd0a6194dd9]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x45fe95)[0x7fd0a6194e95]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x63bb7a)[0x7fd0a6370b7a]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x8914b3)[0x7fd0a65c64b3]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x419e9e)[0x7fd0a614ee9e]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x35dbcb)[0x7fd0a6092bcb]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x35df22)[0x7fd0a6092f22]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x35ec97)[0x7fd0a6093c97]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x4a82d6)[0x7fd0a61dd2d6]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x4a922a)[0x7fd0a61de22a]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0xa77ac2)[0x7fd0a67acac2]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so(+0x8c3eb2)[0x7fd0a65f8eb2]
/lib64/libpthread.so.0(+0x7e65)[0x7fd0a7723e65]
/lib64/libc.so.6(clone+0x6d)[0x7fd0a6e2388d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 103:01 8754845                           /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/bin/java
00600000-00601000 r--p 00000000 103:01 8754845                           /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/bin/java
00601000-00602000 rw-p 00001000 103:01 8754845                           /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/bin/java
00696000-006b7000 rw-p 00000000 00:00 0                                  [heap]
6ce600000-6d5200000 rw-p 00000000 00:00 0 
6d5200000-76f780000 ---p 00000000 00:00 0 
76f780000-7bff00000 rw-p 00000000 00:00 0 
7bff00000-7c0000000 ---p 00000000 00:00 0 
7c0000000-7c02e0000 rw-p 00000000 00:00 0 

Cassandra-stress command was running on aws instances based on ami-03389f1ec374f22d3(eu-north-1) with instance type: c5.2xlarge

Maybe it is related to issue with XFS:

020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !WARNING | kernel: XFS (nvme0n1p1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: XFS (nvme0n1p1): Metadata corruption detected at xfs_inode_buf_verify+0x13f/0x150 [xfs], xfs_inode block 0xbffa40 xfs_inode_buf_verify
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000000: f6 49 e4 f3 00 00 00 00 05 00 00 00 00 00 00 00  .I..............
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000010: a3 9d 00 f8 00 00 00 00 fc 49 e4 f3 00 00 00 00  .........I......
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000020: 05 00 00 00 00 00 00 00 3d 6e 05 f8 96 49 e4 f3  ........=n...I..
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000030: 00 00 00 00 5c 6d e4 d9 9f 49 e4 f3 a2 49 e4 f3  ....\m...I...I..
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000040: 5c c3 ea d9 9b 4b e4 f3 ba 58 e9 d9 0e 4a e4 f3  \....K...X...J..
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000050: ba 58 e9 d9 00 00 00 00 01 00 00 00 00 00 00 00  .X..............
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000060: 36 53 02 f8 01 00 00 00 01 4a e4 f3 00 00 00 00  6S.......J......
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000070: 05 00 00 00 00 00 00 00 a3 9d 00 f8 00 00 00 00  ................
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0xbffa40 len 32 error 117
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !WARNING | kernel: XFS (nvme0n1p1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: XFS (nvme0n1p1): Metadata corruption detected at xfs_inode_buf_verify+0x13f/0x150 [xfs], xfs_inode block 0xbffa40 xfs_inode_buf_verify
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000000: f6 49 e4 f3 00 00 00 00 05 00 00 00 00 00 00 00  .I..............
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000010: a3 9d 00 f8 00 00 00 00 fc 49 e4 f3 00 00 00 00  .........I......
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000020: 05 00 00 00 00 00 00 00 3d 6e 05 f8 96 49 e4 f3  ........=n...I..
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000030: 00 00 00 00 5c 6d e4 d9 9f 49 e4 f3 a2 49 e4 f3  ....\m...I...I..
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000040: 5c c3 ea d9 9b 4b e4 f3 ba 58 e9 d9 0e 4a e4 f3  \....K...X...J..
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000050: ba 58 e9 d9 00 00 00 00 01 00 00 00 00 00 00 00  .X..............
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000060: 36 53 02 f8 01 00 00 00 01 4a e4 f3 00 00 00 00  6S.......J......
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: 00000070: 05 00 00 00 00 00 00 00 a3 9d 00 f8 00 00 00 00  ................
2020-09-07T11:23:41+00:00  longevity-tls-50gb-4d-4-2-loader-node-0a259058-1 !ALERT   | kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0xbffa40 len 32 error 117

all loader log are avaialable: https://cloudius-jenkins-test.s3.amazonaws.com/0a259058-d1db-4a29-933d-7cf34a526f63/20200907_124244/loader-set-0a259058.zip

Full logged output of c-s command: cassandra-stress-l0-c0-k1-545d76a4-c2d0-4f14-bc6f-9224c2739e3d.log

[avikivity]

It looks like that node had memory corruption.

What kernel was running?

If it doesn't reproduce, I recommend closing as a fluke.

[aleksbykov]

Didn't meet it any more for that moment

[avikivity]

Closing as a fluke. Make sure your jdk is fully updated.