load-and-stream: failed to log message: got error format_error (argument not found)

[juliayakovlev]

Installation details
Scylla version (or git commit hash): 4.6.dev-0.20210808.31dcb0d1d with build-id 183c531ab85f2b1ee8010bfa7f29c19cda9098ec
Cluster size:
OS (RHEL/CentOS/Ubuntu/AWS AMI): dtest uses relocatable packages unstable/master:2021-08-08T16:56:08Z

Jenkins dtest-byo job

Error during load_and_stream:

DEBUG 2021-08-10 10:54:21,103 [shard 1] sstable - Writing Summary.db file /data/jenkins/workspace/scylla-master/byo/dtest-byo/scylla/.dtest/dtest-vdnf6_mb/test/node3/data/keyspace1/standard1-4ff4ad30f9c911eb9c7973e09306ac03/md-3-big-Summary.db ERROR 2021-08-10 10:54:21,105 [shard 1] storage_service - service/storage_service.cc:3150 @do_receive: failed to log message: fmt='send_meta_data: got error code={}, from node={}, status={}': fmt::v7::format_error (argument not found)

or

DEBUG 2021-08-10 10:54:21,084 [shard 1] sstable - Writing TOC file /data/jenkins/workspace/scylla-master/byo/dtest-byo/scylla/.dtest/dtest-vdnf6_mb/test/node3/data/keyspace1/standard1-4ff4ad30f9c911eb9c7973e09306ac03/md-3-big-TOC.txt.tmp 
......
DEBUG 2021-08-10 10:54:21,087 [shard 0] flat_mutation_reader - [validator 0x6000000c5d00] partition start:{position: partition_start,null,0}
DEBUG 2021-08-10 10:54:21,087 [shard 0] flat_mutation_reader - [validator 0x6000000c5d00] clustering row:{position: clustered,ckp{},0}
DEBUG 2021-08-10 10:54:21,087 [shard 0] flat_mutation_reader - [validator 0x6000000c5d00] partition end:{position: partition_end,null,0}
DEBUG 2021-08-10 10:54:21,087 [shard 0] flat_mutation_reader - [validator 0x6000000c5d00] EOS
ERROR 2021-08-10 10:54:21,088 [shard 0] storage_service - service/storage_service.cc:3150 @do_receive: failed to log message: fmt='send_meta_data: got error code={}, from node={}, status={}': fmt::v7::format_error (argument not found)
DEBUG 2021-08-10 10:54:21,088 [shard 0] storage_service - send_meta_data: finished reading source from node=127.0.38.1

Log:
https://jenkins.scylladb.com/view/master/job/scylla-master/job/byo/job/dtest-byo/585/artifact/logs-byo.release.2/1628592864668_migration_test.TestLoadAndStream_with_3_0_md.load_and_stream_increase_cluster_with_c_s_data_test/node3.log/*view*/

[slivne]

@asias do we need to backport this to 4.5 ?

[asias]

Yes. we need it.

@avikivity Could you please backport to 4.05. It is a trivial fix to debug log.

[avikivity]

Backported to 4.5. 4.4 does not have this code.