scylladb cluster startup failed: marshaling error: read_simple_bytes - not enough bytes

[MyByte0]

I got a problem. When I build a scylladb cluster with dozens nodes, some node can not start up.
And I restart the nodes, same error.

Installation details
Scylla version (or git commit hash): 5.1.15
Cluster size: 6 nodes in 3 DC, factor 2.
OS (RHEL/CentOS/Ubuntu/AWS AMI): CentOS & x86_64

[MyByte0]

error log:

`
22:00:08 scylla[251390]: [shard 0] storage_proxy - Exception when communicating with 192.168.3.16, to read from system_distributed.service_levels: failed on column system_schema.tables#compression (version: b6240810-eeb7-36d5-9411-43b2d68dddab, id: 4, index: 4, type: org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type))): seastar::internal::backtraced<marshal_exception> (marshaling error: read_simple_bytes - not enough bytes (requested 25237767, got 69) Backtrace: 0x510f8ce 0x510fdc0 0x51100c8 0x23e41c3 0x2413531 0x23b06dd 0x23b03a8 0x2f5ba0b 0x2f58ce6 0x2f621f1 0x2f60734 0x28e29e6 0x4d84b54 0x4d85f37 0x4d8518c 0x4d2b8a8 0x4d2ad81 0x1039cf6 0x10372da /opt/scylladb/libreloc/libc.so.6+0x27b74 0x10361ad
--------
seastar::continuation<seastar::internal::promise_base_with_type<seastar::lw_shared_ptrquery::result_set >, db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68, seastar::futureservice::storage_proxy_coordinator_query_result::then_impl_nrvo<db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68, seastar::future<seastar::lw_shared_ptrquery::result_set > >(db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&&)::{lambda(seastar::internal::promise_base_with_type<seastar::lw_shared_ptrquery::result_set >&&, db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, seastar::future_stateservice::storage_proxy_coordinator_query_result&&)#1}, service::storage_proxy_coordinator_query_result>
--------
seastar::internal::coroutine_traits_base<std::vector<seastar::basic_sstring<char, unsigned int, 15u, true>, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > > >::promise_type
--------
seastar::internal::coroutine_traits_base<std::map<utils::UUID, schema_mutations, std::lessutils::UUID, std::allocator<std::pair<utils::UUID const, schema_mutations> > > >::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::do_with_state<std::tuple<std::vector<canonical_mutation, std::allocator<canonical_mutation> > >, seastar::future >
--------
seastar::continuation<seastar::internal::promise_base_with_type, service::migration_manager::do_merge_schema_from(netw::msg_addr)::$_10, seastar::future::then_impl_nrvo<service::migration_manager::do_merge_schema_from(netw::msg_addr)::$_10, seastar::future >(service::migration_manager::do_merge_schema_from(netw::msg_addr)::$_10&&)::{lambda(seastar::internal::promise_base_with_type&&, service::migration_manager::do_merge_schema_from(netw::msg_addr)::$_10&, seastar::future_stateseastar::internal::monostate&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type, serialized_action::trigger(bool)::{lambda(auto:1&&)#1}, seastar::future::then_wrapped_nrvo<void, serialized_action::trigger(bool)::{lambda(auto:1&&)#1}>(serialized_action::trigger(bool)::{lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type&&, serialized_action::trigger(bool)::{lambda(auto:1&&)#1}&, seastar::future_stateseastar::internal::monostate&&)#1}, void>)

...

22:05:26 scylla[251390]: [shard 0] storage_proxy - Exception when communicating with 192.168.3.41, to read from system_distributed.service_levels: failed on column system_schema.tables#compression (version: b6240810-eeb7-36d5-9411-43b2d68dddab, id: 4, index: 4, type: org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type))): seastar::internal::backtraced<marshal_exception> (marshaling error: read_simple_bytes - not enough bytes (requested 25237767, got 69) Backtrace: 0x510f8ce 0x510fdc0 0x51100c8 0x23e41c3 0x2413531 0x23b06dd 0x23b03a8 0x2f5ba0b 0x2f58ce6 0x2f621f1 0x2f60734 0x28e29e6 0x4d84b54 0x4d85f37 0x4d8518c 0x4d2b8a8 0x4d2ad81 0x1039cf6 0x10372da /opt/scylladb/libreloc/libc.so.6+0x27b74 0x10361ad
--------
seastar::continuation<seastar::internal::promise_base_with_type<seastar::lw_shared_ptrquery::result_set >, db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68, seastar::futureservice::storage_proxy_coordinator_query_result::then_impl_nrvo<db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68, seastar::future<seastar::lw_shared_ptrquery::result_set > >(db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&&)::{lambda(seastar::internal::promise_base_with_type<seastar::lw_shared_ptrquery::result_set >&&, db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, seastar::future_stateservice::storage_proxy_coordinator_query_result&&)#1}, service::storage_proxy_coordinator_query_result>
--------
seastar::internal::coroutine_traits_base<std::vector<seastar::basic_sstring<char, unsigned int, 15u, true>, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > > >::promise_type
--------
seastar::internal::coroutine_traits_base<std::map<utils::UUID, schema_mutations, std::lessutils::UUID, std::allocator<std::pair<utils::UUID const, schema_mutations> > > >::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::do_with_state<std::tuple<std::vector<canonical_mutation, std::allocator<canonical_mutation> > >, seastar::future >
--------
seastar::continuation<seastar::internal::promise_base_with_type, service::migration_manager::do_merge_schema_from(netw::msg_addr)::$_10, seastar::future::then_impl_nrvo<service::migration_manager::do_merge_schema_from(netw::msg_addr)::$_10, seastar::future >(service::migration_manager::do_merge_schema_from(netw::msg_addr)::$_10&&)::{lambda(seastar::internal::promise_base_with_type&&, service::migration_manager::do_merge_schema_from(netw::msg_addr)::$_10&, seastar::future_stateseastar::internal::monostate&&)#1}, void>
--------
seastar::continuation<seastar::internal::promise_base_with_type, serialized_action::trigger(bool)::{lambda(auto:1&&)#1}, seastar::future::then_wrapped_nrvo<void, serialized_action::trigger(bool)::{lambda(auto:1&&)#1}>(serialized_action::trigger(bool)::{lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type&&, serialized_action::trigger(bool)::{lambda(auto:1&&)#1}&, seastar::future_stateseastar::internal::monostate&&)#1}, void>)

...

22:17:03 scylla[139550]: [shard 0] init - Startup failed: query::deserialization_error (failed on column system_schema.tables#compression (version: b6240810-eeb7-36d5-9411-43b2d68dddab, id: 4, index: 4, type: org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type))): seastar::internal::backtraced<marshal_exception> (marshaling error: read_simple_bytes - not enough bytes (requested 25237767, got 69) Backtrace: 0x510f8ce 0x510fdc0 0x51100c8 0x23e41c3 0x2413531 0x23b06dd 0x23b03a8 0x2f5ba0b 0x2f58ce6 0x2f621f1 0x2f60734 0x28e24e6 0x4d84b54 0x4d85f37 0x4d8518c 0x4d2b8a8 0x4d2ad81 0x1039cf6 0x10372da /opt/scylladb/libreloc/libc.so.6+0x27b74 0x10361ad
--------
seastar::continuation<seastar::internal::promise_base_with_type<seastar::lw_shared_ptrquery::result_set >, db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&)::$_67, seastar::futureservice::storage_proxy_coordinator_query_result::then_impl_nrvo<db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&)::$_67, seastar::future<seastar::lw_shared_ptrquery::result_set > >(db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&)::$_67&&)::{lambda(seastar::internal::promise_base_with_type<seastar::lw_shared_ptrquery::result_set >&&, db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&)::$_67&, seastar::future_stateservice::storage_proxy_coordinator_query_result&&)#1}, service::storage_proxy_coordinator_query_result>
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::continuation<seastar::internal::promise_base_with_type, seastar::future::finally_body<seastar::smp::submit_to<seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&&)::{lambda()#1}, false>, seastar::future::then_wrapped_nrvo<seastar::future, seastar::future::finally_body<seastar::smp::submit_to<seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&&)::{lambda()#1}, false> >(seastar::future::finally_body<seastar::smp::submit_to<seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&&)::{lambda()#1}, false>&&)::{lambda(seastar::internal::promise_base_with_type&&, seastar::future::finally_body<seastar::smp::submit_to<seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, auto:1&&)::{lambda()#1}, false>&, seastar::future_stateseastar::internal::monostate&&)#1}, void>))

...

22:27:03 scylla[152826]: [shard 0] init - Startup failed: query::deserialization_error (failed on column system_schema.tables#compression (version: b6240810-eeb7-36d5-9411-43b2d68dddab, id: 4, index: 4, type: org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type))): seastar::internal::backtraced<marshal_exception> (marshaling error: read_simple_bytes - not enough bytes (requested 25237767, got 69) Backtrace: 0x510f8ce 0x510fdc0 0x51100c8 0x23e41c3 0x2413531 0x23b06dd 0x23b03a8 0x2f5ba0b 0x2f58ce6 0x2f621f1 0x2f60734 0x28e24e6 0x4d84b54 0x4d85f37 0x4d8518c 0x4d2b8a8 0x4d2ad81 0x1039cf6 0x10372da /opt/scylladb/libreloc/libc.so.6+0x27b74 0x10361ad
--------
seastar::continuation<seastar::internal::promise_base_with_type<seastar::lw_shared_ptrquery::result_set >, db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&)::$_67, seastar::futureservice::storage_proxy_coordinator_query_result::then_impl_nrvo<db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&)::$_67, seastar::future<seastar::lw_shared_ptrquery::result_set > >(db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&)::$_67&&)::{lambda(seastar::internal::promise_base_with_type<seastar::lw_shared_ptrquery::result_set >&&, db::system_keyspace::query(seastar::shardedservice::storage_proxy&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&)::$_67&, seastar::future_stateservice::storage_proxy_coordinator_query_result&&)#1}, service::storage_proxy_coordinator_query_result>
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::internal::coroutine_traits_base::promise_type
--------
seastar::continuation<seastar::internal::promise_base_with_type, seastar::future::finally_body<seastar::smp::submit_to<seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&&)::{lambda()#1}, false>, seastar::future::then_wrapped_nrvo<seastar::future, seastar::future::finally_body<seastar::smp::submit_to<seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&&)::{lambda()#1}, false> >(seastar::future::finally_body<seastar::smp::submit_to<seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}&&)::{lambda()#1}, false>&&)::{lambda(seastar::internal::promise_base_with_type&&, seastar::future::finally_body<seastar::smp::submit_to<seastar::shardedreplica::database::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future (replica::database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, auto:1&&)::{lambda()#1}, false>&, seastar::future_stateseastar::internal::monostate&&)#1}, void>))
`

[MyByte0]

the backtrace:

[Backtrace #0] void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_3>(seastar::current_backtrace_tasklocal()::$_3&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./seastar/src/util/backtrace.cc:86 seastar::current_tasktrace() at ./build/release/seastar/./seastar/src/util/backtrace.cc:137 seastar::current_backtrace() at ./build/release/seastar/./seastar/src/util/backtrace.cc:170 backtraced<marshal_exception> at ././seastar/include/seastar/util/backtrace.hh:191 (inlined by) std::__exception_ptr::exception_ptr seastar::make_backtraced_exception_ptr<marshal_exception, seastar::basic_sstring<char, unsigned int, 15u, true> >(seastar::basic_sstring<char, unsigned int, 15u, true>&&) at ././seastar/include/seastar/util/backtrace.hh:219 (inlined by) void seastar::throw_with_backtrace<marshal_exception, seastar::basic_sstring<char, unsigned int, 15u, true> >(seastar::basic_sstring<char, unsigned int, 15u, true>&&) at ././seastar/include/seastar/util/backtrace.hh:235 basic_single_fragmented_view<(mutable_view)0> read_simple_bytes<basic_single_fragmented_view<(mutable_view)0> >(basic_single_fragmented_view<(mutable_view)0>&, unsigned long) at ././types.hh:1171 (inlined by) basic_single_fragmented_view<(mutable_view)0> read_collection_value<basic_single_fragmented_view<(mutable_view)0> >(basic_single_fragmented_view<(mutable_view)0>&, cql_serialization_format) at ././types/listlike_partial_deserializing_iterator.hh:39 (inlined by) data_value map_type_impl::deserialize<basic_single_fragmented_view<(mutable_view)0> >(basic_single_fragmented_view<(mutable_view)0>, cql_serialization_format) const at ./types.cc:1158 (anonymous namespace)::deserialize_visitor<basic_single_fragmented_view<(mutable_view)0> >::operator()(map_type_impl const&) at ./types.cc:2188 (inlined by) std::invoke_result<(anonymous namespace)::deserialize_visitor<basic_single_fragmented_view<(mutable_view)0> >, ascii_type_impl const&>::type visit<(anonymous namespace)::deserialize_visitor<basic_single_fragmented_view<(mutable_view)0> > >(abstract_type const&, (anonymous namespace)::deserialize_visitor<basic_single_fragmented_view<(mutable_view)0> >&&) at ././concrete_types.hh:208 data_value abstract_type::deserialize_impl<basic_single_fragmented_view<(mutable_view)0> >(basic_single_fragmented_view<(mutable_view)0>) const at ./types.cc:2201 data_value abstract_type::deserialize<ser::buffer_view<bytes_ostream::fragment_iterator> >(ser::buffer_view<bytes_ostream::fragment_iterator>) const at ././types.hh:504 (inlined by) data_value abstract_type::deserialize_value<ser::buffer_view<bytes_ostream::fragment_iterator> >(ser::buffer_view<bytes_ostream::fragment_iterator>) const at ././types.hh:513 (inlined by) query::result_set_builder::deserialize(query::result_row_view const&, bool) at ./query-result-set.cc:177 query::result_set_builder::accept_new_row(clustering_key_prefix const&, query::result_row_view const&, query::result_row_view const&) at ./query-result-set.cc:97 void query::result_view::consume<query::result_set_builder&>(query::partition_slice const&, query::result_set_builder&) const at ././query-result-reader.hh:177 void query::result_view::consume<query::result_set_builder&>(query::result const&, query::partition_slice const&, query::result_set_builder&) at ././query-result-reader.hh:156 (inlined by) query::result_set::from_raw_result(seastar::lw_shared_ptr<schema const>, query::partition_slice const&, query::result const&) at ./query-result-set.cc:206 operator()<service::storage_proxy_coordinator_query_result> at ./db/system_keyspace.cc:2826 (inlined by) seastar::lw_shared_ptr<query::result_set> std::__invoke_impl<seastar::lw_shared_ptr<query::result_set>, db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result>(std::__invoke_other, db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61 (inlined by) std::__invoke_result<db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result>::type std::__invoke<db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result>(db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:96 (inlined by) std::invoke_result<db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result>::type std::invoke<db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result>(db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result&&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/functional:97 (inlined by) auto seastar::internal::future_invoke<db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result>(db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, service::storage_proxy_coordinator_query_result&&) at ././seastar/include/seastar/core/future.hh:1225 (inlined by) operator() at ././seastar/include/seastar/core/future.hh:1596 (inlined by) void seastar::futurize<seastar::lw_shared_ptr<query::result_set> >::satisfy_with_result_of<seastar::future<service::storage_proxy_coordinator_query_result>::then_impl_nrvo<db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68, seastar::future<seastar::lw_shared_ptr<query::result_set> > >(db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&&)::{lambda(seastar::internal::promise_base_with_type<seastar::lw_shared_ptr<query::result_set> >&&, db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, seastar::future_state<service::storage_proxy_coordinator_query_result>&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::lw_shared_ptr<query::result_set> >&&, db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, seastar::future_state<service::storage_proxy_coordinator_query_result>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::lw_shared_ptr<query::result_set> >&&, db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&&) at ././seastar/include/seastar/core/future.hh:2136 (inlined by) operator() at ././seastar/include/seastar/core/future.hh:1589 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<seastar::lw_shared_ptr<query::result_set> >, db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68, seastar::future<service::storage_proxy_coordinator_query_result>::then_impl_nrvo<db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68, seastar::future<seastar::lw_shared_ptr<query::result_set> > >(db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&&)::{lambda(seastar::internal::promise_base_with_type<seastar::lw_shared_ptr<query::result_set> >&&, db::system_keyspace::query(seastar::sharded<service::storage_proxy>&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&, dht::decorated_key const&, nonwrapping_interval<clustering_key_prefix>)::$_68&, seastar::future_state<service::storage_proxy_coordinator_query_result>&&)#1}, service::storage_proxy_coordinator_query_result>::run_and_dispose() at ././seastar/include/seastar/core/future.hh:781 seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2345 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2752 seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2921 seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2804 seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at ./build/release/seastar/./seastar/src/core/app-template.cc:265 seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at ./build/release/seastar/./seastar/src/core/app-template.cc:156 scylla_main(int, char**) at ./main.cc:555 std::function<int (int, char**)>::operator()(int, char**) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:590 (inlined by) main at ./main.cc:1711

[mykaul]

Looks like some kind of data corruption?
https://opensource.docs.scylladb.com/stable/troubleshooting/sstable-corruption.html might help.

[MyByte0]

Looks like some kind of data corruption? https://opensource.docs.scylladb.com/stable/troubleshooting/sstable-corruption.html might help.

I don't think so. More then 10 nodes get the same problem. they deployed on different machine, use different ssd.

[MyByte0]

Looks like some kind of data corruption? https://opensource.docs.scylladb.com/stable/troubleshooting/sstable-corruption.html might help.

I have tried,it does not work.
I think it is a logic error.The scylladb cluster have 60 nodes, and 20 nodes have the same problem.