Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault during scylla stop with cdc traffic #7821

Closed
aleksbykov opened this issue Dec 22, 2020 · 21 comments
Closed

Segmentation fault during scylla stop with cdc traffic #7821

aleksbykov opened this issue Dec 22, 2020 · 21 comments
Assignees
@bhalevy
Labels
area/cdc bug
Milestone
4.3

Comments

@aleksbykov
Copy link
Contributor

Installation details
Scylla version (or git commit hash): 4.3.rc3-0.20201220.5bd52e4db with build-id ec3427545525638e1720aab150d5addd4b625131
Cluster size: 6
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-0b6d330f7fb62f83f(eu-central-1)
Test-id: 71d77731-dec0-435e-a0dd-100307686018
Job: longevity-cdc-3d-400gb-test
Livemonitor: http://3.125.52.100:3000/d/QJcmf7xMk/scylla-per-server-metrics-nemesis-master?orgId=1&from=now-24h&to=now
instance type: i3.4xlarge

Test cdc-3d-400GB generate dataset with size ~400GB and then start run different nemesis.
For generating dataset and manipulating it used next c-s profiles and commands:

keyspace: cdc_test

keyspace_definition: |

  CREATE KEYSPACE IF NOT EXISTS cdc_test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

table: test_table

table_definition: |

  CREATE TABLE cdc_test.test_table (
    pkid text PRIMARY KEY,
    name text,
    number int,
    starttime timestamp,
    weight decimal,
    steps float,
    t_num tinyint,
    bvalue blob,
    boy boolean,
    vname varchar,
    vage varint,
    nums_set set<int>,
    names_set set<text>
  ) WITH bloom_filter_fp_chance = 0.01
    AND cdc = {'enabled': true, 'preimage': false, 'postimage': false, 'ttl': 3600}
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = 'Request for unit level UIs'
    AND compaction = {'class': 'LeveledCompactionStrategy'}
    AND compression = {'sstable_compression': 'ZstdCompressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

columnspec:
  - name: pkid
    size: fixed(1000)
    population: seq(1..50000000)

  - name: bvalue
    size: fixed(1000)

  - name: vname
    size: fixed(1000)

  - name: nums_set
    size: fixed(5)
    population: exp(1..5)

  - name: names_set
    size: fixed(100)
    population: exp(1..10)

insert:
  partitions: fixed(1)
  batchtype: UNLOGGED

queries:
  read1:
    cql: select * from cdc_test.test_table where pkid = ?
    fields: samerow
  update_name:
    cql: update cdc_test.test_table set name = ?, nums_set = ? where pkid = ?
    fields: samerow
  update_number:
    cql: update cdc_test.test_table set number = ?, names_set = ? where pkid = ?
    fields: samerow
  update_all:
    cql: update cdc_test.test_table set name = ?,number = ?,starttime = ?,weight = ?,steps = ?,t_num = ?,bvalue = ?,boy = ?,vname = ?,vage = ?,nums_set = ?,names_set = ? where pkid = ?
    fields: samerow
  delete1:
    cql: delete from cdc_test.test_table where pkid = ?
    fields: samerow

keyspace: cdc_test

keyspace_definition: |

  CREATE KEYSPACE IF NOT EXISTS cdc_test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

table: test_table_postimage

table_definition: |

  CREATE TABLE cdc_test.test_table_postimage (
    pkid text PRIMARY KEY,
    name text,
    number int,
    starttime timestamp,
    weight decimal,
    steps float,
    t_num tinyint,
    bvalue blob,
    boy boolean,
    vname varchar,
    vage varint,
    nums_set set<int>,
    names_set set<text>
  ) WITH bloom_filter_fp_chance = 0.01
    AND cdc = {'enabled': true, 'preimage': false, 'postimage': true, 'ttl': 3600}
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = 'Request for unit level UIs'
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'SnappyCompressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

columnspec:
  - name: pkid
    size: fixed(1000)
    population: seq(1..50000000)

  - name: bvalue
    size: fixed(1000)

  - name: vname
    size: fixed(1000)

  - name: nums_set
    size: fixed(5)
    population: exp(1..5)

  - name: names_set
    size: fixed(100)
    population: exp(1..10)

insert:
  partitions: fixed(1)
  batchtype: UNLOGGED

queries:
  read1:
    cql: select * from cdc_test.test_table_postimage where pkid = ?
    fields: samerow
  update_name:
    cql: update cdc_test.test_table_postimage set name = ?, nums_set = ? where pkid = ?
    fields: samerow
  update_number:
    cql: update cdc_test.test_table_postimage set number = ?, names_set = ? where pkid = ?
    fields: samerow
  update_all:
    cql: update cdc_test.test_table set name = ?,number = ?,starttime = ?,weight = ?,steps = ?,t_num = ?,bvalue = ?,boy = ?,vname = ?,vage = ?,nums_set = ?,names_set = ? where pkid = ?
    fields: samerow
  delete1:
    cql: delete from cdc_test.test_table_postimage where pkid = ?
    fields: samerow

keyspace: cdc_test

keyspace_definition: |

  CREATE KEYSPACE IF NOT EXISTS cdc_test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

table: test_table_preimage

table_definition: |

  CREATE TABLE cdc_test.test_table_preimage (
    pkid text PRIMARY KEY,
    name text,
    number int,
    starttime timestamp,
    weight decimal,
    steps float,
    t_num tinyint,
    bvalue blob,
    boy boolean,
    vname varchar,
    vage varint,
    nums_set set<int>,
    names_set set<text>
  ) WITH bloom_filter_fp_chance = 0.01
    AND cdc = {'enabled': true, 'preimage': true, 'postimage': false, 'ttl': 3600}
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = 'Request for unit level UIs'
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

columnspec:
  - name: pkid
    size: fixed(1000)
    population: seq(1..50000000)

  - name: bvalue
    size: fixed(1000)

  - name: vname
    size: fixed(1000)

  - name: nums_set
    size: fixed(5)
    population: exp(1..5)

  - name: names_set
    size: fixed(100)
    population: exp(1..10)

insert:
  partitions: fixed(1)
  batchtype: UNLOGGED

queries:
  read1:
    cql: select * from cdc_test.test_table_preimage where pkid = ?
    fields: samerow
  update_name:
    cql: update cdc_test.test_table_preimage set name = ?, nums_set = ? where pkid = ?
    fields: samerow
  update_number:
    cql: update cdc_test.test_table_preimage set number = ?, names_set = ? where pkid = ?
    fields: samerow
  update_all:
    cql: update cdc_test.test_table set name = ?,number = ?,starttime = ?,weight = ?,steps = ?,t_num = ?,bvalue = ?,boy = ?,vname = ?,vage = ?,nums_set = ?,names_set = ? where pkid = ?
    fields: samerow
  delete1:
    cql: delete from cdc_test.test_table_preimage where pkid = ?
    fields: samerow

keyspace: cdc_test

keyspace_definition: |

  CREATE KEYSPACE IF NOT EXISTS cdc_test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

table: test_table_preimage_postimage

table_definition: |

  CREATE TABLE cdc_test.test_table_preimage_postimage (
    pkid text PRIMARY KEY,
    ckid time,
    name text,
    number int,
    starttime timestamp,
    weight decimal,
    steps float,
    t_num tinyint,
    bvalue blob,
    boy boolean,
    vname varchar,
    vage varint,
    nums_set set<int>,
    names_set set<text>
  ) WITH bloom_filter_fp_chance = 0.01
    AND cdc = {'enabled': true, 'preimage': true, 'postimage': true, 'ttl': 3600}
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = 'Request for unit level UIs'
    AND compaction = {'class': 'LeveledCompactionStrategy'}
    AND compression = {'sstable_compression': 'DeflateCompressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

columnspec:
  - name: pkid
    size: fixed(1000)
    population: seq(1..50000000)

  - name: ckid
    cluster: gaussian(2..5)

  - name: bvalue
    size: fixed(1000)

  - name: vname
    size: fixed(1000)

  - name: nums_set
    size: fixed(5)
    population: exp(1..5)

  - name: names_set
    size: fixed(100)
    population: exp(1..10)

insert:
  partitions: fixed(1)
  batchtype: UNLOGGED
  select: gaussian(2..5)/5

queries:
  read1:
    cql: select * from cdc_test.test_table_preimage_postimage where pkid = ?
    fields: samerow
  update_name:
    cql: update cdc_test.test_table_preimage_postimage set name = ?, nums_set = ? where pkid = ?
    fields: samerow
  update_number:
    cql: update cdc_test.test_table_preimage_postimage set number = ?, names_set = ? where pkid = ?
    fields: samerow
  update_all:
    cql: update cdc_test.test_table set name = ?,number = ?,starttime = ?,weight = ?,steps = ?,t_num = ?,bvalue = ?,boy = ?,vname = ?,vage = ?,nums_set = ?,names_set = ? where pkid = ?
    fields: samerow
  delete1:
    cql: delete from cdc_test.test_table_preimage_postimage where pkid = ?
    fields: samerow

C-s commands used to prepare dataset:

"cassandra-stress user no-warmup profile=/tmp/cdc_profile_400gb.yaml ops'(insert=1)' cl=QUORUM n=50000000 -port jmx=6868 -mode cql3 native -rate threads=100",
                    "cassandra-stress user no-warmup profile=/tmp/cdc_profile_400gb_preimage.yaml ops'(insert=1)' cl=QUORUM n=50000000 -port jmx=6868 -mode cql3 native -rate threads=200",
                    "cassandra-stress user no-warmup profile=/tmp/cdc_profile_400gb_postimage.yaml ops'(insert=1)' cl=QUORUM n=50000000 -port jmx=6868 -mode cql3 native -rate threads=200",
                    "cassandra-stress user no-warmup profile=/tmp/cdc_profile_400gb_preimage_postimage.yaml ops'(insert=1)' cl=QUORUM n=50000000 -port jmx=6868 -mode cql3 native -rate threads=200"

C-s commands used to manipulate dataset:

 "cassandra-stress user no-warmup profile=/tmp/cdc_profile_400gb.yaml ops'(insert=2,read1=1,update_number=1,update_name=1,update_all=2,delete1=1)' cl=QUORUM duration=4320m -port jmx=6868 -mode cql3 native -rate threads=100",
              "cassandra-stress user no-warmup profile=/tmp/cdc_profile_400gb_preimage.yaml ops'(insert=2,read1=1,update_number=1,update_name=1,update_all=2,delete1=1)' cl=QUORUM duration=4320m -port jmx=6868 -mode cql3 native -rate threads=100",
              "cassandra-stress user no-warmup profile=/tmp/cdc_profile_400gb_postimage.yaml ops'(insert=2,read1=1,update_number=1,update_name=1,update_all=2,delete1=1)' cl=QUORUM duration=4320m -port jmx=6868 -mode cql3 native -rate threads=100",
              "cassandra-stress user no-warmup profile=/tmp/cdc_profile_400gb_preimage_postimage.yaml ops'(insert=2,read1=1,update_number=1,update_all=2,update_name=1,delete1=1)' cl=QUORUM duration=4320m -port jmx=6868 -mode cql3 native -rate threads=100"

After dataset was generated(this took about ~2hours), nemesis started.
First nemesis run on node6 was MajorCompaction. Command nodetool compact was finished without errors.
Next nemesis RepairStreamingErr. This nemesis should stop scylla, remove several sstables and then start it again.
During scylla stopping, Segmentation fault was detected:

2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | sshd[128947]: pam_unix(sshd:session): session opened for user centos by (uid=0)
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !NOTICE  | sudo: scyllaadm : TTY=unknown ; PWD=/home/scyllaadm ; USER=root ; COMMAND=/bin/systemctl stop scylla-server.service
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | systemd: Stopping Scylla JMX...
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | systemd: Stopped Run Scylla Housekeeping restart mode.
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | systemd: Stopped Run Scylla Housekeeping daily mode.
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !NOTICE  | systemd: scylla-jmx.service: main process exited, code=exited, status=143/n/a
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | systemd: Stopped Scylla JMX.
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !NOTICE  | systemd: Unit scylla-jmx.service entered failed state.
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | systemd: scylla-jmx.service failed.
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | systemd: Stopping Scylla Server...
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] compaction_manager - Stopping 1 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] init - Signal received; shutting down
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] init - Shutting down redis service
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] init - Shutting down redis service was successful
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] init - Shutting down view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 1] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 1] compaction_manager - Stopping 2 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] compaction_manager - Stopping 3 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 1] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 11] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 10] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 13] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 10] compaction_manager - Stopping 4 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 2] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 7] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 13] compaction_manager - Stopping 4 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 2] compaction_manager - Stopping 3 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 10] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 11] compaction_manager - Stopping 3 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 9] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 12] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 11] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 2] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 13] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 4] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 3] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 7] compaction_manager - Stopping 1 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 6] compaction_manager - Asked to stop
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 4] compaction_manager - Stopping 2 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 3] compaction_manager - Stopping 2 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 6] compaction_manager - Stopping 4 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 9] compaction_manager - Stopping 3 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 6] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 9] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 12] compaction_manager - Stopping 4 ongoing compactions
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 4] view - Stopping view builder
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: Segmentation fault on shard 13.
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: Backtrace:
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000002b59122
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000002b597b0
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000002b59a55
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000002b59af0
2020-12-21T22:13:47+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x00007f027193aa8f
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x00000000012a47ae
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x00000000012a5d91
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000000e23049
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000000e236bb
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000000e23b59
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000001178edc
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000001272e25
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000001273738
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x000000000131e3c1
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x000000000133ffdc
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: 0x0000000002df514c
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | kernel: reactor-13[5160]: segfault at 33d ip 00000000012a47af sp 000060d00beff9f0 error 4 in scylla[c2c000+244d000]
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | kernel: Code: 43 38 48 c1 f8 04 4c 01 d8 0f 88 1d 01 00 00 48 83 f8 1f 7e 5e 48 89 c2 48 c1 fa 05 48 89 d1 48 c1 e1 05 48 29 c8 48 c1 e0 04 <48> 03 04 d7 4c 8b 7d 20 4c 89 e5 44 0f b6 73 04 48 2b 28 4d 85 ed
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 12] view - Stopping view builder
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 7] view - Stopping view builder
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 3] view - Stopping view builder
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 8] compaction_manager - Asked to stop
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 8] compaction_manager - Stopping 2 ongoing compactions
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 8] view - Stopping view builder
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] init - Shutting down view builder was successful
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] init - Shutting down local storage
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] storage_service - Drain on shutdown: starts
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] storage_service - Stop transport: starts
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] storage_service - Shutting down native transport
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 1] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error 
system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 6] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 1] cql_server - exception while processing connection: std::system_error (error system:104, sendmsg: Connection reset by peer)
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 4] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 6] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 3] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 7] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 7] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 8] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 1] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 0] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 13] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 11] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 11] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 10] compaction_manager - compaction info: Compaction for cdc_test/test_table_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 6] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 11] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 9] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_preimage_postimage_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 9] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 11] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 11] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 6] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: s
topping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 6] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 12] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 3] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 8] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 9] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 8] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 4] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 3] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 3] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 4] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 6] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 13] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_preimage_postimage_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 10] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 10] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 13] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 6] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_preimage_postimage_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 1] compaction_manager - compaction info: Compaction for cdc_test/test_table was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 1] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 10] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 13] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 10] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 10] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 6] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 6] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 4] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 9] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 9] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 9] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] compaction_manager - compaction info: Compaction for cdc_test/test_table_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:104, sendmsg: Connection reset by peer) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 13] compaction_manager - compaction info: Compaction for cdc_test/test_table_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 2] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 2] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 12] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] compaction_manager - compaction info: Compaction for cdc_test/test_table_preimage_postimage was stopped due to: shutdown: stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 12] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>, seastar::future<void> >
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] compaction_manager - Stopped
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !ERR     | scylla: [shard 12] compaction_manager - compaction failed: seastar::broken_promise (broken promise): stopping
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 6] seastar - Exceptional future ignored: seastar::broken_promise (broken promise), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x131eeb1#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIvEENS_6futureIvE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_EUlOS3_RSZ_ONS_12future_stateINS1_9monostateEEEE_vEE
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !INFO    | scylla: [shard 5] cql_server - exception while processing connection: seastar::nested_exception: std::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::system_error (error system:32, sendmsg: Broken pipe))
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 3] seastar - Exceptional future ignored: seastar::broken_promise (broken promise), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x131eeb1#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   N7seastar12continuationINS_8internal22promise_base_with_typeIvEENS_6futureIvE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_EUlOS3_RSZ_ONS_12future_stateINS1_9monostateEEEE_vEE
...

Decoded segmentation backtrace:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at reactor.cc:?
seastar::print_with_backtrace(seastar::backtrace_buffer&) at reactor.cc:?
 (inlined by) print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:752
seastar::print_with_backtrace(char const*) at reactor.cc:?
void seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at reactor.cc:?
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3466
 (inlined by) _FUN at ./build/release/seastar/./seastar/src/core/reactor.cc:3462
?? ??:0
std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator+=(long) at /usr/include/c++/10/bits/stl_deque.h:240
 (inlined by) std::operator+(std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*> const&, long) at /usr/include/c++/10/bits/stl_deque.h:378
 (inlined by) std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator[](long) const at /usr/include/c++/10/bits/stl_deque.h:252
 (inlined by) std::deque<sstables::compression::segmented_offsets::bucket, std::allocator<sstables::compression::segmented_offsets::bucket> >::operator[](unsigned long) at /usr/include/c++/10/bits/stl_deque.h:1327
 (inlined by) sstables::compression::segmented_offsets::push_back(unsigned long, sstables::compression::segmented_offsets::state&) at ./sstables/compress.cc:214
sstables::compression::segmented_offsets::writer::push_back(unsigned long) at ./sstables/compress.hh:123
 (inlined by) compressed_file_data_sink_impl<crc32_utils, (compressed_checksum_mode)1>::put(seastar::temporary_buffer<char>) at ./sstables/compress.cc:519
seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at table.cc:?
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at ././seastar/include/seastar/core/iostream-impl.hh:432
seastar::output_stream<char>::flush() at table.cc:?
seastar::output_stream<char>::close() at table.cc:?
sstables::file_writer::close() at sstables.cc:?
sstables::mc::writer::~writer() at writer.cc:?
 (inlined by) sstables::mc::writer::~writer() at ./sstables/mx/writer.cc:790
sstables::mc::writer::~writer() at writer.cc:?
flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at compaction.cc:?
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_destroy() at /usr/include/c++/10/optional:260
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_reset() at /usr/include/c++/10/optional:280
 (inlined by) std::_Optional_payload<sstables::compaction_writer, false, false, false>::~_Optional_payload() at /usr/include/c++/10/optional:401
 (inlined by) std::_Optional_base<sstables::compaction_writer, false, false>::~_Optional_base() at /usr/include/c++/10/optional:474
 (inlined by) std::optional<sstables::compaction_writer>::~optional() at /usr/include/c++/10/optional:659
 (inlined by) sstables::compacting_sstable_writer::~compacting_sstable_writer() at ./sstables/compaction.cc:229
 (inlined by) compact_mutation<(emit_only_live_rows)0, (compact_for_sstables)1, sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_mutation() at ././mutation_compactor.hh:468
 (inlined by) compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_for_compaction() at ././mutation_compactor.hh:538
 (inlined by) std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::operator()(compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>*) const at /usr/include/c++/10/bits/unique_ptr.h:85
 (inlined by) std::unique_ptr<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>, std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~unique_ptr() at /usr/include/c++/10/bits/unique_ptr.h:361
 (inlined by) stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::~stable_flattened_mutations_consumer() at ././mutation_reader.hh:342
 (inlined by) flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at ././flat_mutation_reader.hh:201
auto flat_mutation_reader::impl::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:272
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:383
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:389
 (inlined by) seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}::operator()() at ./sstables/compaction.cc:612
 (inlined by) void std::__invoke_impl<void, seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>(std::__invoke_other, seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}&&) at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) std::__invoke_result<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>::type std::__invoke<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>(seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}&&, (seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}&&)...) at /usr/include/c++/10/bits/invoke.h:95
 (inlined by) _ZSt12__apply_implIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEEN7seastar6futureIvEET_ENUl20flat_mutation_readerE_clES8_EUlvE_St5tupleIJEEJEEDcOS7_OT0_St16integer_sequenceImJXspT1_EEE at /usr/include/c++/10/tuple:1723
 (inlined by) _ZSt5applyIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEEN7seastar6futureIvEET_ENUl20flat_mutation_readerE_clES8_EUlvE_St5tupleIJEEEDcOS7_OT0_ at /usr/include/c++/10/tuple:1734
 (inlined by) seastar::future<void> seastar::futurize<void>::apply<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>(seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}&&, std::tuple<>&&) at ././seastar/include/seastar/core/future.hh:2099
 (inlined by) seastar::futurize<std::result_of<std::decay<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>::type ()>::type>::type seastar::async<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>(seastar::thread_attributes, seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}&&, (std::decay<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>&&)...)::{lambda()#1}::operator()() const at ././seastar/include/seastar/core/thread.hh:258
 (inlined by) seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::result_of<std::decay<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>::type ()>::type>::type seastar::async<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>(seastar::thread_attributes, seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}&&, (std::decay<seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}>&&)...)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././seastar/include/seastar/util/noncopyable_function.hh:116
seastar::noncopyable_function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::thread_context::main() at ./build/release/seastar/./seastar/src/core/thread.cc:297

Several decoded backtraces for warning messages:

(DatabaseLogEvent Severity.ERROR): type=DATABASE_ERROR regex=Exception  line_number=53122 node=Node longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 [18.197.96.217 | 10.0.0.66] (seed: False)
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 6] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state, seastar::future >
backtrace > at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::current_backtrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at future.cc:?
seastar::report_failed_future(seastar::future_state_base::any&&) at future.cc:?
seastar::internal::do_with_state, seastar::future >::run_and_dispose() at timestamp_based_splitting_writer.cc:?
 (inlined by) seastar::future_state::clear() at ././seastar/include/seastar/core/future.hh:609
 (inlined by) seastar::future_state::~future_state() at ././seastar/include/seastar/core/future.hh:614
 (inlined by) seastar::future::~future() at ././seastar/include/seastar/core/future.hh:1337
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::bucket_writer::~bucket_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:112
 (inlined by) std::pair::~pair() at /usr/include/c++/10/bits/stl_pair.h:211
 (inlined by) void std::destroy_at >(std::pair*) at /usr/include/c++/10/bits/stl_construct.h:88
 (inlined by) void std::allocator_traits, false> > >::destroy >(std::allocator, false> >&, std::pair*) at /usr/include/c++/10/bits/alloc_traits.h:533
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_node(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2053
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2075
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::clear() at /usr/include/c++/10/bits/hashtable.h:2030
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable() at /usr/include/c++/10/bits/hashtable.h:1387
 (inlined by) std::unordered_map, std::equal_to, std::allocator > >::~unordered_map() at /usr/include/c++/10/bits/unordered_map.h:102
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::~timestamp_based_splitting_mutation_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:109
 (inlined by) std::_Head_base<1ul, mutation_writer::timestamp_based_splitting_mutation_writer, false>::~_Head_base() at /usr/include/c++/10/tuple:124
 (inlined by) std::_Tuple_impl<1ul, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:341
 (inlined by) std::_Tuple_impl<0ul, flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:191
 (inlined by) std::tuple::~tuple() at /usr/include/c++/10/tuple:887
 (inlined by) seastar::internal::do_with_state, seastar::future >::~do_with_state() at ././seastar/include/seastar/core/do_with.hh:37
 (inlined by) seastar::internal::do_with_state, seastar::future >::run_and_dispose() at ././seastar/include/seastar/core/do_with.hh:45
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
seastar::reactor::run_some_tasks() [clone .part.0] at reactor.cc:?
seastar::reactor::run() at reactor.cc:?
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2730
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3906
std::function::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0
DatabaseLogEvent, ERROR, events, DATABASE_ERROR
2020-12-21 22:13:48.000 (DatabaseLogEvent Severity.ERROR): type=DATABASE_ERROR regex=Exception  line_number=53123 node=Node longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 [18.197.96.217 | 10.0.0.66] (seed: False)
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 11] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state, seastar::future >
backtrace > at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::current_backtrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at future.cc:?
seastar::report_failed_future(seastar::future_state_base::any&&) at future.cc:?
seastar::internal::do_with_state, seastar::future >::run_and_dispose() at timestamp_based_splitting_writer.cc:?
 (inlined by) seastar::future_state::clear() at ././seastar/include/seastar/core/future.hh:609
 (inlined by) seastar::future_state::~future_state() at ././seastar/include/seastar/core/future.hh:614
 (inlined by) seastar::future::~future() at ././seastar/include/seastar/core/future.hh:1337
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::bucket_writer::~bucket_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:112
 (inlined by) std::pair::~pair() at /usr/include/c++/10/bits/stl_pair.h:211
 (inlined by) void std::destroy_at >(std::pair*) at /usr/include/c++/10/bits/stl_construct.h:88
 (inlined by) void std::allocator_traits, false> > >::destroy >(std::allocator, false> >&, std::pair*) at /usr/include/c++/10/bits/alloc_traits.h:533
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_node(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2053
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2075
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::clear() at /usr/include/c++/10/bits/hashtable.h:2030
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable() at /usr/include/c++/10/bits/hashtable.h:1387
 (inlined by) std::unordered_map, std::equal_to, std::allocator > >::~unordered_map() at /usr/include/c++/10/bits/unordered_map.h:102
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::~timestamp_based_splitting_mutation_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:109
 (inlined by) std::_Head_base<1ul, mutation_writer::timestamp_based_splitting_mutation_writer, false>::~_Head_base() at /usr/include/c++/10/tuple:124
 (inlined by) std::_Tuple_impl<1ul, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:341
 (inlined by) std::_Tuple_impl<0ul, flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:191
 (inlined by) std::tuple::~tuple() at /usr/include/c++/10/tuple:887
 (inlined by) seastar::internal::do_with_state, seastar::future >::~do_with_state() at ././seastar/include/seastar/core/do_with.hh:37
 (inlined by) seastar::internal::do_with_state, seastar::future >::run_and_dispose() at ././seastar/include/seastar/core/do_with.hh:45
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
seastar::reactor::run_some_tasks() [clone .part.0] at reactor.cc:?
seastar::reactor::run() at reactor.cc:?
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2730
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3906
std::function::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0
DatabaseLogEvent, ERROR, events, DATABASE_ERROR
2020-12-21 22:13:48.000 (DatabaseLogEvent Severity.ERROR): type=DATABASE_ERROR regex=Exception  line_number=53124 node=Node longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 [18.197.96.217 | 10.0.0.66] (seed: False)
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 9] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_preimage_postimage_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state, seastar::future >
backtrace > at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::current_backtrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at future.cc:?
seastar::report_failed_future(seastar::future_state_base::any&&) at future.cc:?
seastar::internal::do_with_state, seastar::future >::run_and_dispose() at timestamp_based_splitting_writer.cc:?
 (inlined by) seastar::future_state::clear() at ././seastar/include/seastar/core/future.hh:609
 (inlined by) seastar::future_state::~future_state() at ././seastar/include/seastar/core/future.hh:614
 (inlined by) seastar::future::~future() at ././seastar/include/seastar/core/future.hh:1337
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::bucket_writer::~bucket_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:112
 (inlined by) std::pair::~pair() at /usr/include/c++/10/bits/stl_pair.h:211
 (inlined by) void std::destroy_at >(std::pair*) at /usr/include/c++/10/bits/stl_construct.h:88
 (inlined by) void std::allocator_traits, false> > >::destroy >(std::allocator, false> >&, std::pair*) at /usr/include/c++/10/bits/alloc_traits.h:533
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_node(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2053
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2075
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::clear() at /usr/include/c++/10/bits/hashtable.h:2030
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable() at /usr/include/c++/10/bits/hashtable.h:1387
 (inlined by) std::unordered_map, std::equal_to, std::allocator > >::~unordered_map() at /usr/include/c++/10/bits/unordered_map.h:102
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::~timestamp_based_splitting_mutation_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:109
 (inlined by) std::_Head_base<1ul, mutation_writer::timestamp_based_splitting_mutation_writer, false>::~_Head_base() at /usr/include/c++/10/tuple:124
 (inlined by) std::_Tuple_impl<1ul, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:341
 (inlined by) std::_Tuple_impl<0ul, flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:191
 (inlined by) std::tuple::~tuple() at /usr/include/c++/10/tuple:887
 (inlined by) seastar::internal::do_with_state, seastar::future >::~do_with_state() at ././seastar/include/seastar/core/do_with.hh:37
 (inlined by) seastar::internal::do_with_state, seastar::future >::run_and_dispose() at ././seastar/include/seastar/core/do_with.hh:45
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
seastar::reactor::run_some_tasks() [clone .part.0] at reactor.cc:?
seastar::reactor::run() at reactor.cc:?
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2730
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3906
std::function::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0
DatabaseLogEvent, ERROR, events, DATABASE_ERROR
2020-12-21 22:13:48.000 (DatabaseLogEvent Severity.ERROR): type=DATABASE_ERROR regex=Exception  line_number=53132 node=Node longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 [18.197.96.217 | 10.0.0.66] (seed: False)
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 3] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state, seastar::future >
backtrace > at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::current_backtrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at future.cc:?
seastar::report_failed_future(seastar::future_state_base::any&&) at future.cc:?
seastar::internal::do_with_state, seastar::future >::run_and_dispose() at timestamp_based_splitting_writer.cc:?
 (inlined by) seastar::future_state::clear() at ././seastar/include/seastar/core/future.hh:609
 (inlined by) seastar::future_state::~future_state() at ././seastar/include/seastar/core/future.hh:614
 (inlined by) seastar::future::~future() at ././seastar/include/seastar/core/future.hh:1337
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::bucket_writer::~bucket_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:112
 (inlined by) std::pair::~pair() at /usr/include/c++/10/bits/stl_pair.h:211
 (inlined by) void std::destroy_at >(std::pair*) at /usr/include/c++/10/bits/stl_construct.h:88
 (inlined by) void std::allocator_traits, false> > >::destroy >(std::allocator, false> >&, std::pair*) at /usr/include/c++/10/bits/alloc_traits.h:533
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_node(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2053
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2075
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::clear() at /usr/include/c++/10/bits/hashtable.h:2030
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable() at /usr/include/c++/10/bits/hashtable.h:1387
 (inlined by) std::unordered_map, std::equal_to, std::allocator > >::~unordered_map() at /usr/include/c++/10/bits/unordered_map.h:102
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::~timestamp_based_splitting_mutation_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:109
 (inlined by) std::_Head_base<1ul, mutation_writer::timestamp_based_splitting_mutation_writer, false>::~_Head_base() at /usr/include/c++/10/tuple:124
 (inlined by) std::_Tuple_impl<1ul, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:341
 (inlined by) std::_Tuple_impl<0ul, flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:191
 (inlined by) std::tuple::~tuple() at /usr/include/c++/10/tuple:887
 (inlined by) seastar::internal::do_with_state, seastar::future >::~do_with_state() at ././seastar/include/seastar/core/do_with.hh:37
 (inlined by) seastar::internal::do_with_state, seastar::future >::run_and_dispose() at ././seastar/include/seastar/core/do_with.hh:45
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
seastar::reactor::run_some_tasks() [clone .part.0] at reactor.cc:?
seastar::reactor::run() at reactor.cc:?
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2730
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3906
std::function::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0
DatabaseLogEvent, ERROR, events, DATABASE_ERROR
2020-12-21 22:13:48.000 (DatabaseLogEvent Severity.ERROR): type=DATABASE_ERROR regex=Exception  line_number=53141 node=Node longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 [18.197.96.217 | 10.0.0.66] (seed: False)
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 13] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_preimage_postimage_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state, seastar::future >
backtrace > at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::current_backtrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at future.cc:?
seastar::report_failed_future(seastar::future_state_base::any&&) at future.cc:?
seastar::internal::do_with_state, seastar::future >::run_and_dispose() at timestamp_based_splitting_writer.cc:?
 (inlined by) seastar::future_state::clear() at ././seastar/include/seastar/core/future.hh:609
 (inlined by) seastar::future_state::~future_state() at ././seastar/include/seastar/core/future.hh:614
 (inlined by) seastar::future::~future() at ././seastar/include/seastar/core/future.hh:1337
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::bucket_writer::~bucket_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:112
 (inlined by) std::pair::~pair() at /usr/include/c++/10/bits/stl_pair.h:211
 (inlined by) void std::destroy_at >(std::pair*) at /usr/include/c++/10/bits/stl_construct.h:88
 (inlined by) void std::allocator_traits, false> > >::destroy >(std::allocator, false> >&, std::pair*) at /usr/include/c++/10/bits/alloc_traits.h:533
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_node(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2053
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2075
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::clear() at /usr/include/c++/10/bits/hashtable.h:2030
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable() at /usr/include/c++/10/bits/hashtable.h:1387
 (inlined by) std::unordered_map, std::equal_to, std::allocator > >::~unordered_map() at /usr/include/c++/10/bits/unordered_map.h:102
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::~timestamp_based_splitting_mutation_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:109
 (inlined by) std::_Head_base<1ul, mutation_writer::timestamp_based_splitting_mutation_writer, false>::~_Head_base() at /usr/include/c++/10/tuple:124
 (inlined by) std::_Tuple_impl<1ul, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:341
 (inlined by) std::_Tuple_impl<0ul, flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:191
 (inlined by) std::tuple::~tuple() at /usr/include/c++/10/tuple:887
 (inlined by) seastar::internal::do_with_state, seastar::future >::~do_with_state() at ././seastar/include/seastar/core/do_with.hh:37
 (inlined by) seastar::internal::do_with_state, seastar::future >::run_and_dispose() at ././seastar/include/seastar/core/do_with.hh:45
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
seastar::reactor::run_some_tasks() [clone .part.0] at reactor.cc:?
seastar::reactor::run() at reactor.cc:?
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2730
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3906
std::function::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0
DatabaseLogEvent, ERROR, events, DATABASE_ERROR
2020-12-21 22:13:48.000 (DatabaseLogEvent Severity.ERROR): type=DATABASE_ERROR regex=Exception  line_number=53163 node=Node longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 [18.197.96.217 | 10.0.0.66] (seed: False)
2020-12-21T22:13:48+00:00  longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 !WARNING | scylla: [shard 2] seastar - Exceptional future ignored: sstables::compaction_stop_exception (Compaction for cdc_test/test_table_scylla_cdc_log was stopped due to: shutdown), backtrace:    0x2fb5b0d#012   0x2fb5d60#012   0x2fb61c9#012   0x2b017d7#012   0x2b0191c#012   0x255e738#012   0x2b57567#012   0x2b5781e#012   0x2b97105#012   0x2ba88ba#012   0x2b2148d#012   /opt/scylladb/libreloc/libpthread.so.0+0x9431#012   /opt/scylladb/libreloc/libc.so.6+0x101912#012   --------#012   seastar::internal::do_with_state, seastar::future >
backtrace > at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./seastar/src/util/backtrace.cc:86
seastar::current_tasktrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::current_backtrace() at /usr/include/c++/10/bits/basic_string.h:323
seastar::report_failed_future(std::__exception_ptr::exception_ptr const&) at future.cc:?
seastar::report_failed_future(seastar::future_state_base::any&&) at future.cc:?
seastar::internal::do_with_state, seastar::future >::run_and_dispose() at timestamp_based_splitting_writer.cc:?
 (inlined by) seastar::future_state::clear() at ././seastar/include/seastar/core/future.hh:609
 (inlined by) seastar::future_state::~future_state() at ././seastar/include/seastar/core/future.hh:614
 (inlined by) seastar::future::~future() at ././seastar/include/seastar/core/future.hh:1337
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::bucket_writer::~bucket_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:112
 (inlined by) std::pair::~pair() at /usr/include/c++/10/bits/stl_pair.h:211
 (inlined by) void std::destroy_at >(std::pair*) at /usr/include/c++/10/bits/stl_construct.h:88
 (inlined by) void std::allocator_traits, false> > >::destroy >(std::allocator, false> >&, std::pair*) at /usr/include/c++/10/bits/alloc_traits.h:533
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_node(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2053
 (inlined by) std::__detail::_Hashtable_alloc, false> > >::_M_deallocate_nodes(std::__detail::_Hash_node, false>*) at /usr/include/c++/10/bits/hashtable_policy.h:2075
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::clear() at /usr/include/c++/10/bits/hashtable.h:2030
 (inlined by) std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable() at /usr/include/c++/10/bits/hashtable.h:1387
 (inlined by) std::unordered_map, std::equal_to, std::allocator > >::~unordered_map() at /usr/include/c++/10/bits/unordered_map.h:102
 (inlined by) mutation_writer::timestamp_based_splitting_mutation_writer::~timestamp_based_splitting_mutation_writer() at ./mutation_writer/timestamp_based_splitting_writer.cc:109
 (inlined by) std::_Head_base<1ul, mutation_writer::timestamp_based_splitting_mutation_writer, false>::~_Head_base() at /usr/include/c++/10/tuple:124
 (inlined by) std::_Tuple_impl<1ul, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:341
 (inlined by) std::_Tuple_impl<0ul, flat_mutation_reader, mutation_writer::timestamp_based_splitting_mutation_writer>::~_Tuple_impl() at /usr/include/c++/10/tuple:191
 (inlined by) std::tuple::~tuple() at /usr/include/c++/10/tuple:887
 (inlined by) seastar::internal::do_with_state, seastar::future >::~do_with_state() at ././seastar/include/seastar/core/do_with.hh:37
 (inlined by) seastar::internal::do_with_state, seastar::future >::run_and_dispose() at ././seastar/include/seastar/core/do_with.hh:45
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at reactor.cc:?
seastar::reactor::run_some_tasks() [clone .part.0] at reactor.cc:?
seastar::reactor::run() at reactor.cc:?
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2730
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3906
std::function::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

Nodes:

| longevity-cdc-3d-400gb-4-3-db-node-71d77731-1      | eu-central-1a | 18.156.69.205  | 71d77731-dec0-435e-a0dd-100307686018 |
| longevity-cdc-3d-400gb-4-3-db-node-71d77731-2      | eu-central-1a | 18.195.178.196 | 71d77731-dec0-435e-a0dd-100307686018 |
| longevity-cdc-3d-400gb-4-3-db-node-71d77731-3      | eu-central-1a | 18.194.209.38  | 71d77731-dec0-435e-a0dd-100307686018 |
| longevity-cdc-3d-400gb-4-3-db-node-71d77731-5      | eu-central-1a | 3.120.188.14   | 71d77731-dec0-435e-a0dd-100307686018 |
| longevity-cdc-3d-400gb-4-3-db-node-71d77731-6      | eu-central-1a | 18.197.96.217  | 71d77731-dec0-435e-a0dd-100307686018 |
@slivne slivne added this to the 4.3 milestone Dec 22, 2020
@bhalevy
Copy link
Member

bhalevy commented Dec 22, 2020

Is there a core dump?

@aleksbykov
Copy link
Contributor Author

yes, It was not uploaded yet. And stay on node:

ssh -i ~/.ssh/scylla-qa-ec2 centos@18.197.96.217
[scyllaadm@longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 ~]$ ls /var/lib/scylla/coredump/
core.scylla.997.c3d94c27822143d4be9355039c0bb9ee.5131.1608588828000000  core.scylla.997.c3d94c27822143d4be9355039c0bb9ee.5131.1608588828000000.gz
[scyllaadm@longevity-cdc-3d-400gb-4-3-db-node-71d77731-6 ~]$

@aleksbykov
Copy link
Contributor Author

Coredump could be downloaded: https://storage.cloud.google.com/upload.scylladb.com/7821/core.scylla.997.c3d94c27822143d4be9355039c0bb9ee.5131.1608588828000000.gz

@bhalevy
Copy link
Member

bhalevy commented Dec 22, 2020

In https://github.com/scylladb/scylla/blob/5bd52e4dbaee7d67f5f3ab19e54dc84d935104e3/sstables/compress.cc#L214
state looks corrupt, possibly overwritten as it contains a pointer in _current_index

(gdb) fr 4
#4  sstables::compression::segmented_offsets::push_back (this=0x60d00a9711e8, offset=2394450, s=...) at sstables/compress.cc:214
214	    const uint64_t bucket_base_offset = _storage[s._current_bucket_index].base_offset;
(gdb) p/x s
$4 = (sstables::compression::segmented_offsets::state &) @0x60d00182fbf8: {
  _current_index = 0x60d001cee0d4,
  _current_bucket_index = 0xc17,
  _current_bucket_segment_index = 0x5f1e1a9,
  _current_segment_relative_index = 0x84,
  _current_segment_offset_bits = 0x0
}
(gdb) p &s
$5 = (sstables::compression::segmented_offsets::state *) 0x60d00182fbf8
(gdb) scylla ptr 0x60d001cee0d4
thread 1, small (size <= 10240), free (0x60d001cec800 +6356)

@bhalevy
Copy link
Member

bhalevy commented Dec 22, 2020

Hmm, s._current_index equals this->_size

(gdb) p/x this->_size
$11 = 0x60d001cee0d4

@bhalevy
Copy link
Member

bhalevy commented Dec 22, 2020 

(gdb) fr
#11 0x0000000001178edd in sstables::file_writer::close (this=0x60d00f621240) at sstables/sstables.cc:942
942	        _out.close().get();
(gdb) p _filename._M_payload._M_payload._M_value
$32 = "/var/lib/scylla/data/cdc_test/test_table_preimage_postimage_scylla_cdc_log-586fab30439a11eb8694", '0' <repeats 11 times>, "8/md-4773-big-Data.db"

@bhalevy
Copy link
Member

bhalevy commented Dec 22, 2020 •
edited
Loading 

(gdb) fr
#6  compressed_file_data_sink_impl<crc32_utils, (compressed_checksum_mode)1>::put (this=0x60d00182fb60, buf=...) at sstables/compress.cc:519
(gdb) p	buf
$34 = {
  _buffer = 0x60d001e1f000 "j1`U\037@\016\033TM\035rLfYJ\017\064\036\031\006\001;\017\064r\f[\005DRQP\v\004_pw)F\020up\f\"",
  _size = 2616,
  _deleter = {
    _impl = 0x60d001e1f001
  }
}

That is ok, as deleter::from_raw_object returns reinterpret_cast<impl*>(x | 1)

@fgelcer fgelcer mentioned this issue Dec 22, 2020
Closed
@bhalevy
Copy link
Member

bhalevy commented Dec 23, 2020 

(gdb) fr 6
#6  compressed_file_data_sink_impl<crc32_utils, (compressed_checksum_mode)1>::put (this=0x60d00182fb60, buf=...) at sstables/compress.cc:519
519	        _offsets.push_back(_pos);
(gdb) p _compression_metadata
$24 = (sstables::compression *) 0x60d00a971188
(gdb) scylla ptr 0x60d00a971188
thread 1, small (size <= 640), free (0x60d00a971180 +8)

@bhalevy
Copy link
Member

bhalevy commented Dec 23, 2020 

#13 sstables::mc::writer::~writer (this=0x60d0001c2a00, __in_chrg=<optimized out>) at sstables/mx/writer.cc:790
790	    close_writer(_data_writer);
(gdb) p &_sst
$30 = (sstables::sstable *) 0x60d00d9c9200
(gdb) scylla ptr 0x60d00d9c9200
thread 1, small (size <= 768), free (0x60d00d9c9200 +0)

So the mc::writer's sstable's _sst._components->compression is used after after free.
We get a pointer to it in https://github.com/scylladb/scylla/blob/5bd52e4dbaee7d67f5f3ab19e54dc84d935104e3/sstables/mx/writer.cc#L843-L848

@bhalevy
Copy link
Member

bhalevy commented Dec 23, 2020

looks like compaction is also used after free (after it was stopped?) in ~compacting_sstable_writer

(gdb) up
#21 sstables::compacting_sstable_writer::~compacting_sstable_writer (this=0x60d0003034c8, __in_chrg=<optimized out>) at sstables/compaction.cc:229
229	class compacting_sstable_writer {
(gdb) p *this
$31 = {
  _c = @0x60d0001a2d00,
  _compaction_writer = {
    <std::_Optional_base<sstables::compaction_writer, false, false>> = {
      <std::_Optional_base_impl<sstables::compaction_writer, std::_Optional_base<sstables::compaction_writer, false, false> >> = {<No data fields>}, 
      members of std::_Optional_base<sstables::compaction_writer, false, false>:
      _M_payload = {
        <std::_Optional_payload<sstables::compaction_writer, true, false, false>> = {
          <std::_Optional_payload_base<sstables::compaction_writer>> = {
            _M_payload = {
              _M_empty = {<No data fields>},
              _M_value = {
                writer = {
                  _impl = {
                    _M_t = {
                      <std::__uniq_ptr_impl<sstables::sstable_writer::writer_impl, std::default_delete<sstables::sstable_writer::writer_impl> >> = {
                        _M_t = {
                          <std::_Tuple_impl<0, sstables::sstable_writer::writer_impl*, std::default_delete<sstables::sstable_writer::writer_impl> >> = {
                            <std::_Tuple_impl<1, std::default_delete<sstables::sstable_writer::writer_impl> >> = {
                              <std::_Head_base<1, std::default_delete<sstables::sstable_writer::writer_impl>, true>> = {
                                <std::default_delete<sstables::sstable_writer::writer_impl>> = {<No data fields>}, <No data fields>}, <No data fields>}, 
                            <std::_Head_base<0, sstables::sstable_writer::writer_impl*, false>> = {
                              _M_head_impl = 0x60d0001c2a00
                            }, <No data fields>}, <No data fields>}
                      }, <No data fields>}
                  }
                },
                monitor = {
                  _M_t = {
                    <std::__uniq_ptr_impl<sstables::compaction_write_monitor, std::default_delete<sstables::compaction_write_monitor> >> = {
                      _M_t = {
                        <std::_Tuple_impl<0, sstables::compaction_write_monitor*, std::default_delete<sstables::compaction_write_monitor> >> = {
                          <std::_Tuple_impl<1, std::default_delete<sstables::compaction_write_monitor> >> = {
                            <std::_Head_base<1, std::default_delete<sstables::compaction_write_monitor>, true>> = {
                              <std::default_delete<sstables::compaction_write_monitor>> = {<No data fields>}, <No data fields>}, <No data fields>}, 
                          <std::_Head_base<0, sstables::compaction_write_monitor*, false>> = {
                            _M_head_impl = 0x60d000a98600
                          }, <No data fields>}, <No data fields>}
                    }, <No data fields>}
                },
                sst = {
                  _p = 0x60d00d9c9200
                }
              }
            },
            _M_engaged = false
          }, <No data fields>}, <No data fields>}
    }, 
    <std::_Enable_copy_move<false, false, true, true, std::optional<sstables::compaction_writer> >> = {<No data fields>}, <No data fields>}
}
(gdb) scylla ptr 0x60d0001a2d00
thread 1, small (size <= 1280), free (0x60d0001a2d00 +0)
(gdb) scylla ptr 0x60d0001c2a00
thread 1, small (size <= 1792), live (0x60d0001c2a00 +0)
(gdb) scylla ptr 0x60d000a98600
thread 1, small (size <= 64), live (0x60d000a98600 +0)
(gdb) p *(sstables::compaction_write_monitor*)0x60d000a98600
warning: RTTI symbol not found for class 'seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::handle_exception<sstables::remove_by_toc_name(std::basic_string_view<char, std::char_traits<char> >)::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(bool)#2}::operator()(bool) const::{lambda(seastar::file&)#1}::operator()({lambda(bool)#2}) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::input_stream<char>&)#1}::operator()({lambda(unsigned long)#1}) const::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::input_stream<char>) const::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>) const::{lambda(std::__exception_ptr::exception_ptr)#1}>(sstables::remove_by_toc_name(std::basic_string_view<char, std::char_traits<char> >)::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(bool)#2}::operator()(bool) const::{lambda(seastar::file&)#1}::operator()({lambda(bool)#2}) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::input_stream<char>&)#1}::operator()({lambda(unsigned long)#1}) const::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::input_stream<char>) const::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>) const::{lambda(std::__exception_ptr::exception_ptr)#1}&&)::{lambda({lambda(seastar::temporary_buffer<char>)#1})#1}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, {lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}>({lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>'
$32 = warning: RTTI symbol not found for class 'seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::handle_exception<sstables::remove_by_toc_name(std::basic_string_view<char, std::char_traits<char> >)::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(bool)#2}::operator()(bool) const::{lambda(seastar::file&)#1}::operator()({lambda(bool)#2}) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::input_stream<char>&)#1}::operator()({lambda(unsigned long)#1}) const::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::input_stream<char>) const::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>) const::{lambda(std::__exception_ptr::exception_ptr)#1}>(sstables::remove_by_toc_name(std::basic_string_view<char, std::char_traits<char> >)::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(bool)#2}::operator()(bool) const::{lambda(seastar::file&)#1}::operator()({lambda(bool)#2}) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::input_stream<char>&)#1}::operator()({lambda(unsigned long)#1}) const::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::input_stream<char>) const::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>) const::{lambda(std::__exception_ptr::exception_ptr)#1}&&)::{lambda({lambda(seastar::temporary_buffer<char>)#1})#1}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, {lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}>({lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>'
{
  <sstables::write_monitor> = {
    _vptr.write_monitor = 0x400128 <vtable for seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::handle_exception<sstables::remove_by_toc_name(std::basic_string_view<char, std::char_traits<char> >)::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(bool)#2}::operator()(bool) const::{lambda(seastar::file&)#1}::operator()({lambda(bool)#2}) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::input_stream<char>&)#1}::operator()({lambda(unsigned long)#1}) const::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::input_stream<char>) const::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>) const::{lambda(std::__exception_ptr::exception_ptr)#1}>(sstables::remove_by_toc_name(std::basic_string_view<char, std::char_traits<char> >)::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(bool)#2}::operator()(bool) const::{lambda(seastar::file&)#1}::operator()({lambda(bool)#2}) const::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda(seastar::input_stream<char>&)#1}::operator()({lambda(unsigned long)#1}) const::{lambda(seastar::temporary_buffer<char>)#1}::operator()(seastar::input_stream<char>) const::{lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}::operator()(seastar::basic_sstring<char, unsigned int, 15u, true>) const::{lambda(std::__exception_ptr::exception_ptr)#1}&&)::{lambda({lambda(seastar::temporary_buffer<char>)#1})#1}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, {lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}>({lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::basic_sstring<char, unsigned int, 15u, true>)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>+16>
  }, 
  <backlog_write_progress_manager> = {
    _vptr.backlog_write_progress_manager = 0x3
  }, 
  members of sstables::compaction_write_monitor:
  _sst = {
    _p = 0x1
  },
  _cf = @0x60d008cb3fd0,
  _tracker = 0x60d008cb3fd8,
  _progress_seen = 0,
  _maximum_timestamp = 106446487574608,
  _sstable_level = 104
}

@bhalevy
Copy link
Member

bhalevy commented Dec 23, 2020

My theory is that the unhandled exception might be the root cause of the use-after free since timestamp_based_splitting_mutation_writer::abort() is not waited on by feed_writer()

@gleb-cloudius does that makes sense?
Can you please review this patch that I sent to the ML "mutation_writer: splitting writers: abort: handle consumer exceptions"?

@raphaelsc
Copy link
Member

yesterday, when going through this issue, it was clear to me that it's a use-after-free, with twcs interposer, where sstable object is used after freed when accessed by sst writer dtor.

@bhalevy
Copy link
Member

bhalevy commented Dec 23, 2020

yesterday, when going through this issue, it was clear to me that it's a use-after-free, with twcs interposer, where sstable object is used after freed when accessed by sst writer dtor.

We discussed on the ML.
It looks like after 61520a3, we (correctly) no longer call wr.consume_end_of_stream() on the error path, leading to the use after free.

@avikivity suggested this:

I think the problem here was that _consume_fut was not forwarded if
consume_end_of_stream() was not called.

I think the correct approach is to make feed_writer() unconditionally
call Writer::close(). This gives Writer a single place to do cleanup -
whether needed due to abort(), or due to an exception in one of the
Writer::consume() callbacks.

For the splitting writers, the close() would call each of the underlying
writer's close() and wait for them.

I'm working on reproducing the issue and cooking a patch to fix it.

@raphaelsc
Copy link
Member

raphaelsc commented Dec 23, 2020 •
edited
Loading

yesterday, when going through this issue, it was clear to me that it's a use-after-free, with twcs interposer, where sstable object is used after freed when accessed by sst writer dtor.

We discussed on the ML.
It looks like after 61520a3, we (correctly) no longer call wr.consume_end_of_stream() on the error path, leading to the use after free.

@avikivity suggested this:

I think the problem here was that _consume_fut was not forwarded if
consume_end_of_stream() was not called.
I think the correct approach is to make feed_writer() unconditionally
call Writer::close(). This gives Writer a single place to do cleanup -
whether needed due to abort(), or due to an exception in one of the
Writer::consume() callbacks.
For the splitting writers, the close() would call each of the underlying
writer's close() and wait for them.

I'm working on reproducing the issue and cooking a patch to fix it.

it also seems to me that Gleb's patch exposed this bug. I think the problem is in destruction order of compaction_writer, where sstable is destroyed before sstable writer, and the latter relies on the former, and because the latter takes a reference to the former and accesses it on its dtor, we end up with an use-after-free, which only happens on exception because otherwise sstable_writer is "finished" via an explicit call.

we can fix this issue by making compaction_writer automatic destruction safe:

 struct compaction_writer {
+    shared_sstable sst;
     // We use a ptr for pointer stability and so that it can be null
     // when using a noop monitor.
     sstable_writer writer;
     // The order in here is important. A monitor must be destroyed before the writer it is monitoring since it has a
     // periodic timer that checks the writer.
     std::unique_ptr<compaction_write_monitor> monitor;
-    shared_sstable sst;

@raphaelsc
Copy link
Member

raphaelsc commented Dec 23, 2020 •
edited
Loading

It's funny how we fixed a similar dtor ordering bug here before, where monitor should be destroyed before sstable_writer, but we missed the fact that sstable_writer should also be destroyed before sstable object.

@bhalevy
Copy link
Member

bhalevy commented Dec 23, 2020

we can fix this issue by making compaction_writer automatic destruction safe:

That's a good idea regardless of calling close from feed_writer

@bhalevy
Copy link
Member

bhalevy commented Dec 23, 2020

@raphaelsc to prevent similar issue in the future why not keep a shared_sstable in the writer rather than a reference in sstable_writer::writer_impl?

@raphaelsc
Copy link
Member

raphaelsc commented Dec 23, 2020 via email

@aleksbykov aleksbykov mentioned this issue Dec 28, 2020
Open
@roydahan
Copy link

@slivne do we have it marked as blocker for 4.3?

bhalevy added a commit to bhalevy/scylla that referenced this issue Dec 28, 2020
@bhalevy
sstables: sstable_writer: keep a shared_sstable
77a06c5 
Rather than a sstable& as we already hit cases
where the sstable is destructed before the writer
a couple times due to implicit destruction ordering.

Refs scylladb#7821

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
@bhalevy
Copy link
Member

bhalevy commented Dec 29, 2020

@avikivity please backport to all live release.
(4.1 and 4.2 will have a merge conflict that looks obvious to resolve)

avikivity pushed a commit that referenced this issue Dec 29, 2020
@bhalevy @avikivity
compaction: compaction_writer: destroy shared_sstable after the sstab…
a366de2 
…le_writer

sstable_writer may depend on the sstable throughout its whole lifecycle.
If the sstable is freed before the sstable_writer we might hit use-after-free
as in the follwing case:
```
std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator+=(long) at /usr/include/c++/10/bits/stl_deque.h:240
 (inlined by) std::operator+(std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*> const&, long) at /usr/include/c++/10/bits/stl_deque.h:378
 (inlined by) std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator[](long) const at /usr/include/c++/10/bits/stl_deque.h:252
 (inlined by) std::deque<sstables::compression::segmented_offsets::bucket, std::allocator<sstables::compression::segmented_offsets::bucket> >::operator[](unsigned long) at /usr/include/c++/10/bits/stl_deque.h:1327
 (inlined by) sstables::compression::segmented_offsets::push_back(unsigned long, sstables::compression::segmented_offsets::state&) at ./sstables/compress.cc:214
sstables::compression::segmented_offsets::writer::push_back(unsigned long) at ./sstables/compress.hh:123
 (inlined by) compressed_file_data_sink_impl<crc32_utils, (compressed_checksum_mode)1>::put(seastar::temporary_buffer<char>) at ./sstables/compress.cc:519
seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at table.cc:?
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at ././seastar/include/seastar/core/iostream-impl.hh:432
seastar::output_stream<char>::flush() at table.cc:?
seastar::output_stream<char>::close() at table.cc:?
sstables::file_writer::close() at sstables.cc:?
sstables::mc::writer::~writer() at writer.cc:?
 (inlined by) sstables::mc::writer::~writer() at ./sstables/mx/writer.cc:790
sstables::mc::writer::~writer() at writer.cc:?
flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at compaction.cc:?
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_destroy() at /usr/include/c++/10/optional:260
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_reset() at /usr/include/c++/10/optional:280
 (inlined by) std::_Optional_payload<sstables::compaction_writer, false, false, false>::~_Optional_payload() at /usr/include/c++/10/optional:401
 (inlined by) std::_Optional_base<sstables::compaction_writer, false, false>::~_Optional_base() at /usr/include/c++/10/optional:474
 (inlined by) std::optional<sstables::compaction_writer>::~optional() at /usr/include/c++/10/optional:659
 (inlined by) sstables::compacting_sstable_writer::~compacting_sstable_writer() at ./sstables/compaction.cc:229
 (inlined by) compact_mutation<(emit_only_live_rows)0, (compact_for_sstables)1, sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_mutation() at ././mutation_compactor.hh:468
 (inlined by) compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_for_compaction() at ././mutation_compactor.hh:538
 (inlined by) std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::operator()(compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>*) const at /usr/include/c++/10/bits/unique_ptr.h:85
 (inlined by) std::unique_ptr<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>, std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~unique_ptr() at /usr/include/c++/10/bits/unique_ptr.h:361
 (inlined by) stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::~stable_flattened_mutations_consumer() at ././mutation_reader.hh:342
 (inlined by) flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at ././flat_mutation_reader.hh:201
auto flat_mutation_reader::impl::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:272
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:383
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:389
 (inlined by) seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}::operator()() at ./sstables/compaction.cc:612
```

What happens here is that:

    compressed_file_data_sink_impl(output_stream<char> out, sstables::compression* cm, sstables::local_compression lc)
            : _out(std::move(out))
            , _compression_metadata(cm)
            , _offsets(_compression_metadata->offsets.get_writer())
            , _compression(lc)
            , _full_checksum(ChecksumType::init_checksum())

_compression_metadata points to a buffer held by the sstable object.
and _compression_metadata->offsets.get_writer returns a writer that keeps
a reference to the segmented_offsets in the sstables::compression
that is used in the ~writer -> close path.

Fixes #7821

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20201227145726.33319-1-bhalevy@scylladb.com>
(cherry picked from commit 8a745a0)
avikivity pushed a commit that referenced this issue Dec 29, 2020
@bhalevy @avikivity
compaction: compaction_writer: destroy shared_sstable after the sstab…
c2af59a 
…le_writer

sstable_writer may depend on the sstable throughout its whole lifecycle.
If the sstable is freed before the sstable_writer we might hit use-after-free
as in the follwing case:
```
std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator+=(long) at /usr/include/c++/10/bits/stl_deque.h:240
 (inlined by) std::operator+(std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*> const&, long) at /usr/include/c++/10/bits/stl_deque.h:378
 (inlined by) std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator[](long) const at /usr/include/c++/10/bits/stl_deque.h:252
 (inlined by) std::deque<sstables::compression::segmented_offsets::bucket, std::allocator<sstables::compression::segmented_offsets::bucket> >::operator[](unsigned long) at /usr/include/c++/10/bits/stl_deque.h:1327
 (inlined by) sstables::compression::segmented_offsets::push_back(unsigned long, sstables::compression::segmented_offsets::state&) at ./sstables/compress.cc:214
sstables::compression::segmented_offsets::writer::push_back(unsigned long) at ./sstables/compress.hh:123
 (inlined by) compressed_file_data_sink_impl<crc32_utils, (compressed_checksum_mode)1>::put(seastar::temporary_buffer<char>) at ./sstables/compress.cc:519
seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at table.cc:?
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at ././seastar/include/seastar/core/iostream-impl.hh:432
seastar::output_stream<char>::flush() at table.cc:?
seastar::output_stream<char>::close() at table.cc:?
sstables::file_writer::close() at sstables.cc:?
sstables::mc::writer::~writer() at writer.cc:?
 (inlined by) sstables::mc::writer::~writer() at ./sstables/mx/writer.cc:790
sstables::mc::writer::~writer() at writer.cc:?
flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at compaction.cc:?
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_destroy() at /usr/include/c++/10/optional:260
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_reset() at /usr/include/c++/10/optional:280
 (inlined by) std::_Optional_payload<sstables::compaction_writer, false, false, false>::~_Optional_payload() at /usr/include/c++/10/optional:401
 (inlined by) std::_Optional_base<sstables::compaction_writer, false, false>::~_Optional_base() at /usr/include/c++/10/optional:474
 (inlined by) std::optional<sstables::compaction_writer>::~optional() at /usr/include/c++/10/optional:659
 (inlined by) sstables::compacting_sstable_writer::~compacting_sstable_writer() at ./sstables/compaction.cc:229
 (inlined by) compact_mutation<(emit_only_live_rows)0, (compact_for_sstables)1, sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_mutation() at ././mutation_compactor.hh:468
 (inlined by) compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_for_compaction() at ././mutation_compactor.hh:538
 (inlined by) std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::operator()(compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>*) const at /usr/include/c++/10/bits/unique_ptr.h:85
 (inlined by) std::unique_ptr<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>, std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~unique_ptr() at /usr/include/c++/10/bits/unique_ptr.h:361
 (inlined by) stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::~stable_flattened_mutations_consumer() at ././mutation_reader.hh:342
 (inlined by) flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at ././flat_mutation_reader.hh:201
auto flat_mutation_reader::impl::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:272
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:383
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:389
 (inlined by) seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}::operator()() at ./sstables/compaction.cc:612
```

What happens here is that:

    compressed_file_data_sink_impl(output_stream<char> out, sstables::compression* cm, sstables::local_compression lc)
            : _out(std::move(out))
            , _compression_metadata(cm)
            , _offsets(_compression_metadata->offsets.get_writer())
            , _compression(lc)
            , _full_checksum(ChecksumType::init_checksum())

_compression_metadata points to a buffer held by the sstable object.
and _compression_metadata->offsets.get_writer returns a writer that keeps
a reference to the segmented_offsets in the sstables::compression
that is used in the ~writer -> close path.

Fixes #7821

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20201227145726.33319-1-bhalevy@scylladb.com>
(cherry picked from commit 8a745a0)
avikivity pushed a commit that referenced this issue Dec 29, 2020
@bhalevy @avikivity
compaction: compaction_writer: destroy shared_sstable after the sstab…
e878e31 
…le_writer

sstable_writer may depend on the sstable throughout its whole lifecycle.
If the sstable is freed before the sstable_writer we might hit use-after-free
as in the follwing case:
```
std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator+=(long) at /usr/include/c++/10/bits/stl_deque.h:240
 (inlined by) std::operator+(std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*> const&, long) at /usr/include/c++/10/bits/stl_deque.h:378
 (inlined by) std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator[](long) const at /usr/include/c++/10/bits/stl_deque.h:252
 (inlined by) std::deque<sstables::compression::segmented_offsets::bucket, std::allocator<sstables::compression::segmented_offsets::bucket> >::operator[](unsigned long) at /usr/include/c++/10/bits/stl_deque.h:1327
 (inlined by) sstables::compression::segmented_offsets::push_back(unsigned long, sstables::compression::segmented_offsets::state&) at ./sstables/compress.cc:214
sstables::compression::segmented_offsets::writer::push_back(unsigned long) at ./sstables/compress.hh:123
 (inlined by) compressed_file_data_sink_impl<crc32_utils, (compressed_checksum_mode)1>::put(seastar::temporary_buffer<char>) at ./sstables/compress.cc:519
seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at table.cc:?
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at ././seastar/include/seastar/core/iostream-impl.hh:432
seastar::output_stream<char>::flush() at table.cc:?
seastar::output_stream<char>::close() at table.cc:?
sstables::file_writer::close() at sstables.cc:?
sstables::mc::writer::~writer() at writer.cc:?
 (inlined by) sstables::mc::writer::~writer() at ./sstables/mx/writer.cc:790
sstables::mc::writer::~writer() at writer.cc:?
flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at compaction.cc:?
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_destroy() at /usr/include/c++/10/optional:260
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_reset() at /usr/include/c++/10/optional:280
 (inlined by) std::_Optional_payload<sstables::compaction_writer, false, false, false>::~_Optional_payload() at /usr/include/c++/10/optional:401
 (inlined by) std::_Optional_base<sstables::compaction_writer, false, false>::~_Optional_base() at /usr/include/c++/10/optional:474
 (inlined by) std::optional<sstables::compaction_writer>::~optional() at /usr/include/c++/10/optional:659
 (inlined by) sstables::compacting_sstable_writer::~compacting_sstable_writer() at ./sstables/compaction.cc:229
 (inlined by) compact_mutation<(emit_only_live_rows)0, (compact_for_sstables)1, sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_mutation() at ././mutation_compactor.hh:468
 (inlined by) compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_for_compaction() at ././mutation_compactor.hh:538
 (inlined by) std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::operator()(compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>*) const at /usr/include/c++/10/bits/unique_ptr.h:85
 (inlined by) std::unique_ptr<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>, std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~unique_ptr() at /usr/include/c++/10/bits/unique_ptr.h:361
 (inlined by) stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::~stable_flattened_mutations_consumer() at ././mutation_reader.hh:342
 (inlined by) flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at ././flat_mutation_reader.hh:201
auto flat_mutation_reader::impl::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:272
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:383
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:389
 (inlined by) seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}::operator()() at ./sstables/compaction.cc:612
```

What happens here is that:

    compressed_file_data_sink_impl(output_stream<char> out, sstables::compression* cm, sstables::local_compression lc)
            : _out(std::move(out))
            , _compression_metadata(cm)
            , _offsets(_compression_metadata->offsets.get_writer())
            , _compression(lc)
            , _full_checksum(ChecksumType::init_checksum())

_compression_metadata points to a buffer held by the sstable object.
and _compression_metadata->offsets.get_writer returns a writer that keeps
a reference to the segmented_offsets in the sstables::compression
that is used in the ~writer -> close path.

Fixes #7821

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20201227145726.33319-1-bhalevy@scylladb.com>
(cherry picked from commit 8a745a0)
@avikivity
Copy link
Member

Backported to 4.1, 4.2, 4.3.

bhalevy added a commit to bhalevy/scylla that referenced this issue Dec 30, 2020
@bhalevy
sstables: sstable_writer: keep a shared_sstable
b6d7c4d 
Rather than a sstable& as we already hit cases
where the sstable is destructed before the writer
a couple times due to implicit destruction ordering.

Refs scylladb#7821

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Dec 31, 2020
@bhalevy
sstables: sstable_writer: keep a shared_sstable
6b2fa33 
Rather than a sstable& as we already hit cases
where the sstable is destructed before the writer
a couple times due to implicit destruction ordering.

Refs scylladb#7821

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
avikivity pushed a commit that referenced this issue Jan 4, 2021
@bhalevy @avikivity
compaction: compaction_writer: destroy shared_sstable after the sstab…
918ec5e 
…le_writer

sstable_writer may depend on the sstable throughout its whole lifecycle.
If the sstable is freed before the sstable_writer we might hit use-after-free
as in the follwing case:
```
std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator+=(long) at /usr/include/c++/10/bits/stl_deque.h:240
 (inlined by) std::operator+(std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*> const&, long) at /usr/include/c++/10/bits/stl_deque.h:378
 (inlined by) std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator[](long) const at /usr/include/c++/10/bits/stl_deque.h:252
 (inlined by) std::deque<sstables::compression::segmented_offsets::bucket, std::allocator<sstables::compression::segmented_offsets::bucket> >::operator[](unsigned long) at /usr/include/c++/10/bits/stl_deque.h:1327
 (inlined by) sstables::compression::segmented_offsets::push_back(unsigned long, sstables::compression::segmented_offsets::state&) at ./sstables/compress.cc:214
sstables::compression::segmented_offsets::writer::push_back(unsigned long) at ./sstables/compress.hh:123
 (inlined by) compressed_file_data_sink_impl<crc32_utils, (compressed_checksum_mode)1>::put(seastar::temporary_buffer<char>) at ./sstables/compress.cc:519
seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at table.cc:?
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at ././seastar/include/seastar/core/iostream-impl.hh:432
seastar::output_stream<char>::flush() at table.cc:?
seastar::output_stream<char>::close() at table.cc:?
sstables::file_writer::close() at sstables.cc:?
sstables::mc::writer::~writer() at writer.cc:?
 (inlined by) sstables::mc::writer::~writer() at ./sstables/mx/writer.cc:790
sstables::mc::writer::~writer() at writer.cc:?
flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at compaction.cc:?
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_destroy() at /usr/include/c++/10/optional:260
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_reset() at /usr/include/c++/10/optional:280
 (inlined by) std::_Optional_payload<sstables::compaction_writer, false, false, false>::~_Optional_payload() at /usr/include/c++/10/optional:401
 (inlined by) std::_Optional_base<sstables::compaction_writer, false, false>::~_Optional_base() at /usr/include/c++/10/optional:474
 (inlined by) std::optional<sstables::compaction_writer>::~optional() at /usr/include/c++/10/optional:659
 (inlined by) sstables::compacting_sstable_writer::~compacting_sstable_writer() at ./sstables/compaction.cc:229
 (inlined by) compact_mutation<(emit_only_live_rows)0, (compact_for_sstables)1, sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_mutation() at ././mutation_compactor.hh:468
 (inlined by) compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_for_compaction() at ././mutation_compactor.hh:538
 (inlined by) std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::operator()(compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>*) const at /usr/include/c++/10/bits/unique_ptr.h:85
 (inlined by) std::unique_ptr<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>, std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~unique_ptr() at /usr/include/c++/10/bits/unique_ptr.h:361
 (inlined by) stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::~stable_flattened_mutations_consumer() at ././mutation_reader.hh:342
 (inlined by) flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at ././flat_mutation_reader.hh:201
auto flat_mutation_reader::impl::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:272
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:383
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:389
 (inlined by) seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}::operator()() at ./sstables/compaction.cc:612
```

What happens here is that:

    compressed_file_data_sink_impl(output_stream<char> out, sstables::compression* cm, sstables::local_compression lc)
            : _out(std::move(out))
            , _compression_metadata(cm)
            , _offsets(_compression_metadata->offsets.get_writer())
            , _compression(lc)
            , _full_checksum(ChecksumType::init_checksum())

_compression_metadata points to a buffer held by the sstable object.
and _compression_metadata->offsets.get_writer returns a writer that keeps
a reference to the segmented_offsets in the sstables::compression
that is used in the ~writer -> close path.

Fixes #7821

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20201227145726.33319-1-bhalevy@scylladb.com>
(cherry picked from commit 8a745a0)
avikivity pushed a commit that referenced this issue Jan 4, 2021
@bhalevy @avikivity
compaction: compaction_writer: destroy shared_sstable after the sstab…
e05e7b2 
…le_writer

sstable_writer may depend on the sstable throughout its whole lifecycle.
If the sstable is freed before the sstable_writer we might hit use-after-free
as in the follwing case:
```
std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator+=(long) at /usr/include/c++/10/bits/stl_deque.h:240
 (inlined by) std::operator+(std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*> const&, long) at /usr/include/c++/10/bits/stl_deque.h:378
 (inlined by) std::_Deque_iterator<sstables::compression::segmented_offsets::bucket, sstables::compression::segmented_offsets::bucket&, sstables::compression::segmented_offsets::bucket*>::operator[](long) const at /usr/include/c++/10/bits/stl_deque.h:252
 (inlined by) std::deque<sstables::compression::segmented_offsets::bucket, std::allocator<sstables::compression::segmented_offsets::bucket> >::operator[](unsigned long) at /usr/include/c++/10/bits/stl_deque.h:1327
 (inlined by) sstables::compression::segmented_offsets::push_back(unsigned long, sstables::compression::segmented_offsets::state&) at ./sstables/compress.cc:214
sstables::compression::segmented_offsets::writer::push_back(unsigned long) at ./sstables/compress.hh:123
 (inlined by) compressed_file_data_sink_impl<crc32_utils, (compressed_checksum_mode)1>::put(seastar::temporary_buffer<char>) at ./sstables/compress.cc:519
seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at table.cc:?
 (inlined by) seastar::output_stream<char>::put(seastar::temporary_buffer<char>) at ././seastar/include/seastar/core/iostream-impl.hh:432
seastar::output_stream<char>::flush() at table.cc:?
seastar::output_stream<char>::close() at table.cc:?
sstables::file_writer::close() at sstables.cc:?
sstables::mc::writer::~writer() at writer.cc:?
 (inlined by) sstables::mc::writer::~writer() at ./sstables/mx/writer.cc:790
sstables::mc::writer::~writer() at writer.cc:?
flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at compaction.cc:?
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_destroy() at /usr/include/c++/10/optional:260
 (inlined by) std::_Optional_payload_base<sstables::compaction_writer>::_M_reset() at /usr/include/c++/10/optional:280
 (inlined by) std::_Optional_payload<sstables::compaction_writer, false, false, false>::~_Optional_payload() at /usr/include/c++/10/optional:401
 (inlined by) std::_Optional_base<sstables::compaction_writer, false, false>::~_Optional_base() at /usr/include/c++/10/optional:474
 (inlined by) std::optional<sstables::compaction_writer>::~optional() at /usr/include/c++/10/optional:659
 (inlined by) sstables::compacting_sstable_writer::~compacting_sstable_writer() at ./sstables/compaction.cc:229
 (inlined by) compact_mutation<(emit_only_live_rows)0, (compact_for_sstables)1, sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_mutation() at ././mutation_compactor.hh:468
 (inlined by) compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>::~compact_for_compaction() at ././mutation_compactor.hh:538
 (inlined by) std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::operator()(compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>*) const at /usr/include/c++/10/bits/unique_ptr.h:85
 (inlined by) std::unique_ptr<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer>, std::default_delete<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~unique_ptr() at /usr/include/c++/10/bits/unique_ptr.h:361
 (inlined by) stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >::~stable_flattened_mutations_consumer() at ././mutation_reader.hh:342
 (inlined by) flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >::~consumer_adapter() at ././flat_mutation_reader.hh:201
auto flat_mutation_reader::impl::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:272
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter>(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, flat_mutation_reader::no_filter, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:383
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> > >(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer, noop_compacted_fragments_consumer> >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ././flat_mutation_reader.hh:389
 (inlined by) seastar::future<void> sstables::compaction::setup<noop_compacted_fragments_consumer>(noop_compacted_fragments_consumer)::{lambda(flat_mutation_reader)#1}::operator()(flat_mutation_reader)::{lambda()#1}::operator()() at ./sstables/compaction.cc:612
```

What happens here is that:

    compressed_file_data_sink_impl(output_stream<char> out, sstables::compression* cm, sstables::local_compression lc)
            : _out(std::move(out))
            , _compression_metadata(cm)
            , _offsets(_compression_metadata->offsets.get_writer())
            , _compression(lc)
            , _full_checksum(ChecksumType::init_checksum())

_compression_metadata points to a buffer held by the sstable object.
and _compression_metadata->offsets.get_writer returns a writer that keeps
a reference to the segmented_offsets in the sstables::compression
that is used in the ~writer -> close path.

Fixes #7821

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20201227145726.33319-1-bhalevy@scylladb.com>
(cherry picked from commit 8a745a0)
@avikivity avikivity mentioned this issue Aug 26, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bhalevy bhalevy

Labels
area/cdc bug
Projects
None yet
Milestone
4.3
Development

No branches or pull requests
7 participants
@avikivity @raphaelsc @slivne @bhalevy @roydahan @scylladb-promoter @aleksbykov