fio 3.33 report Invalid or incomplete multibyte or wide character #1517

xirs · 2023-02-14T06:41:03Z

Please acknowledge the following before creating a ticket

I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug:

Environment:

fio version:

Reproduction steps

xirs · 2023-02-14T06:43:36Z

Environment:
CentOS Linux release 7.4.1708
3.10.0-693.21.7.el7.x86_64

fio version:
3.33 (newest currenly)

Reproduction steps:
execute fio as below, just write to a local file:
fio-3.33 -name=fiofiotest -ioengine=libaio -direct=1 -verify=crc32 -thread -size=8G -offset=0 -time_based -numjobs=1 -bs=4K -iodepth=64 -rw=rw -randrepeat=0 -group_reporting -serialize_overlap 1 -filename=/export/core-file/fiotest123 -runtime=$((3600*24))

it will report a error shortly
[root@A06-R08-I132-3-614X44X fiotest]# fio-3.33 -name=fiofiotest -ioengine=libaio -direct=1 -verify=crc32 -thread -size=8G -offset=0 -time_based -numjobs=1 -bs=4K -iodepth=64 -rw=rw -randrepeat=0 -group_reporting -serialize_overlap 1 -filename=/export/core-file/fiotest123 -runtime=$((3600*24))
fiofiotest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 thread
verify: bad header rand_seed 1845770608904403392, wanted 4656945441063251280 at file /export/core-file/fiotest123 offset 4096, length 4096 (requested block: offset=4096, length=4096)
verify: bad header rand_seed 7675794325821844708, wanted 3848311307460056390 at file /export/core-file/fiotest123 offset 8192, length 4096 (requested block: offset=8192, length=4096)
verify: bad header rand_seed 457404114628905840, wanted 1845770608904403392 at file /export/core-file/fiotest123 offset 12288, length 4096 (requested block: offset=12288, length=4096)
verify: bad header rand_seed 6232983246036802550, wanted 208498272759195835 at file /export/core-file/fiotest123 offset 16384, length 4096 (requested block: offset=16384, length=4096)
verify: bad header rand_seed 10779570665355886058, wanted 118310454829890910 at file /export/core-file/fiotest123 offset 28672, length 4096 (requested block: offset=28672, length=4096)
verify: bad header rand_seed 7286359895752155750, wanted 5856131477148689880 at file /export/core-file/fiotest123 offset 57344, length 4096 (requested block: offset=57344, length=4096)
verify: bad header rand_seed 1996027472099509160, wanted 7242134130413744160 at file /export/core-file/fiotest123 offset 24576, length 4096 (requested block: offset=24576, length=4096)
verify: bad header rand_seed 11701504227202758010, wanted 3853672836711140398 at file /export/core-file/fiotest123 offset 40960, length 4096 (requested block: offset=40960, length=4096)
verify: bad header rand_seed 8308781121096274, wanted 1727624096829885457 at file /export/core-file/fiotest123 offset 61440, length 4096 (requested block: offset=61440, length=4096)
verify: bad header rand_seed 10382531219288108397, wanted 6232983246036802550 at file /export/core-file/fiotest123 offset 53248, length 4096 (requested block: offset=53248, length=4096)
verify: bad header rand_seed 8532681277354264188, wanted 3982263580057451198 at file /export/core-file/fiotest123 offset 32768, length 4096 (requested block: offset=32768, length=4096)
verify: bad header rand_seed 100525657264688800, wanted 4430946663470163144 at file /export/core-file/fiotest123 offset 98304, length 4096 (requested block: offset=98304, length=4096)
verify: bad header rand_seed 3322477544768529200, wanted 1830224276857590600 at file /export/core-file/fiotest123 offset 126976, length 4096 (requested block: offset=126976, length=4096)
verify: bad header rand_seed 14260855121351490822, wanted 1462862487343059768 at file /export/core-file/fiotest123 offset 143360, length 4096 (requested block: offset=143360, length=4096)
verify: bad header rand_seed 1794924216015461300, wanted 1996027472099509160 at file /export/core-file/fiotest123 offset 81920, length 4096 (requested block: offset=81920, length=4096)
verify: bad header rand_seed 7100998290563970884, wanted 6774288366088297444 at file /export/core-file/fiotest123 offset 73728, length 4096 (requested block: offset=73728, length=4096)
verify: bad header rand_seed 497629195766708565, wanted 3241964680274168640 at file /export/core-file/fiotest123 offset 77824, length 4096 (requested block: offset=77824, length=4096)
verify: bad header rand_seed 6008069816980022470, wanted 7286359895752155750 at file /export/core-file/fiotest123 offset 163840, length 4096 (requested block: offset=163840, length=4096)
verify: bad header rand_seed 282301576315547703, wanted 1733600205484746090 at file /export/core-file/fiotest123 offset 102400, length 4096 (requested block: offset=102400, length=4096)
verify: bad header rand_seed 4950096981892584622, wanted 11701504227202758010 at file /export/core-file/fiotest123 offset 110592, length 4096 (requested block: offset=110592, length=4096)
verify: bad header rand_seed 5321296336587953428, wanted 3802265734256920167 at file /export/core-file/fiotest123 offset 176128, length 4096 (requested block: offset=176128, length=4096)
verify: bad header rand_seed 3241964680274168640, wanted 7675794325821844708 at file /export/core-file/fiotest123 offset 20480, length 4096 (requested block: offset=20480, length=4096)
verify: bad header rand_seed 6945192814491124077, wanted 7952730004257171600 at file /export/core-file/fiotest123 offset 114688, length 4096 (requested block: offset=114688, length=4096)
verify: bad header rand_seed 2751929237733793416, wanted 1693260321665426593 at file /export/core-file/fiotest123 offset 135168, length 4096 (requested block: offset=135168, length=4096)
verify: bad header rand_seed 14910152530430073364, wanted 1794924216015461300 at file /export/core-file/fiotest123 offset 200704, length 4096 (requested block: offset=200704, length=4096)
verify: bad header rand_seed 1462862487343059768, wanted 3974683700872848352 at file /export/core-file/fiotest123 offset 49152, length 4096 (requested block: offset=49152, length=4096)
verify: bad header rand_seed 979389102072117120, wanted 11839060728924768577 at file /export/core-file/fiotest123 offset 151552, length 4096 (requested block: offset=151552, length=4096)
verify: bad header rand_seed 16430069281790300591, wanted 1672437951892672176 at file /export/core-file/fiotest123 offset 45056, length 4096 (requested block: offset=45056, length=4096)
verify: bad header rand_seed 2147182415651703259, wanted 8308781121096274 at file /export/core-file/fiotest123 offset 167936, length 4096 (requested block: offset=167936, length=4096)
verify: bad header rand_seed 981316703657560816, wanted 6155463757778286237 at file /export/core-file/fiotest123 offset 147456, length 4096 (requested block: offset=147456, length=4096)
verify: bad header rand_seed 10816363619074398880, wanted 8022198666215861184 at file /export/core-file/fiotest123 offset 229376, length 4096 (requested block: offset=229376, length=4096)
verify: bad header rand_seed 1107637059645586008, wanted 8948455488376579660 at file /export/core-file/fiotest123 offset 192512, length 4096 (requested block: offset=192512, length=4096)
verify: bad header rand_seed 3802265734256920167, wanted 1322700767255217160 at file /export/core-file/fiotest123 offset 69632, length 4096 (requested block: offset=69632, length=4096)
verify: bad header rand_seed 5302791621682633888, wanted 2577671191612467522 at file /export/core-file/fiotest123 offset 245760, length 4096 (requested block: offset=245760, length=4096)
verify: bad header rand_seed 1473410412951278750, wanted 7100998290563970884 at file /export/core-file/fiotest123 offset 180224, length 4096 (requested block: offset=180224, length=4096)
verify: bad header rand_seed 292621132717039838, wanted 1088636669440839913 at file /export/core-file/fiotest123 offset 208896, length 4096 (requested block: offset=208896, length=4096)
verify: bad header rand_seed 3627472129630975912, wanted 104979830088410108 at file /export/core-file/fiotest123 offset 94208, length 4096 (requested block: offset=94208, length=4096)
verify: bad header rand_seed 11074800191867490702, wanted 10779570665355886058 at file /export/core-file/fiotest123 offset 86016, length 4096 (requested block: offset=86016, length=4096)
verify: bad header rand_seed 186975604673134354, wanted 285305010997995092 at file /export/core-file/fiotest123 offset 188416, length 4096 (requested block: offset=188416, length=4096)
verify: bad header rand_seed 1969023118549458400, wanted 4268553994388312182 at file /export/core-file/fiotest123 offset 106496, length 4096 (requested block: offset=106496, length=4096)
verify: bad header rand_seed 1896157488945279051, wanted 4186192194837855576 at file /export/core-file/fiotest123 offset 204800, length 4096 (requested block: offset=204800, length=4096)
verify: bad header rand_seed 10601912755718265716, wanted 16430069281790300591 at file /export/core-file/fiotest123 offset 118784, length 4096 (requested block: offset=118784, length=4096)
verify: bad header rand_seed 943683090860702628, wanted 482306165939872323 at file /export/core-file/fiotest123 offset 221184, length 4096 (requested block: offset=221184, length=4096)
verify: bad header rand_seed 4676374691188900435, wanted 3627472129630975912 at file /export/core-file/fiotest123 offset 233472, length 4096 (requested block: offset=233472, length=4096)
verify: bad header rand_seed 195788669885463879, wanted 3502439119431801289 at file /export/core-file/fiotest123 offset 225280, length 4096 (requested block: offset=225280, length=4096)
verify: bad header rand_seed 3271917326586969249, wanted 2969992747944951625 at file /export/core-file/fiotest123 offset 131072, length 4096 (requested block: offset=131072, length=4096)
verify: bad header rand_seed 5406333618482947059, wanted 5242097001380581224 at file /export/core-file/fiotest123 offset 237568, length 4096 (requested block: offset=237568, length=4096)
verify: bad header rand_seed 1000442517899799576, wanted 3622991068606401943 at file /export/core-file/fiotest123 offset 155648, length 4096 (requested block: offset=155648, length=4096)
verify: bad header rand_seed 463650874646953800, wanted 613287760329901216 at file /export/core-file/fiotest123 offset 249856, length 4096 (requested block: offset=249856, length=4096)
verify: bad header rand_seed 7396566665270905566, wanted 3038120751606011540 at file /export/core-file/fiotest123 offset 253952, length 4096 (requested block: offset=253952, length=4096)
verify: bad header rand_seed 2062545215934996480, wanted 2840348381712047686 at file /export/core-file/fiotest123 offset 139264, length 4096 (requested block: offset=139264, length=4096)
verify: bad header rand_seed 8174947392626881088, wanted 16334828667629311866 at file /export/core-file/fiotest123 offset 172032, length 4096 (requested block: offset=172032, length=4096)
verify: bad header rand_seed 1719893049609622924, wanted 10382531219288108397 at file /export/core-file/fiotest123 offset 159744, length 4096 (requested block: offset=159744, length=4096)
verify: bad header rand_seed 2641401711809209104, wanted 497629195766708565 at file /export/core-file/fiotest123 offset 184320, length 4096 (requested block: offset=184320, length=4096)
verify: bad header rand_seed 7707049195397541988, wanted 764909874126869656 at file /export/core-file/fiotest123 offset 217088, length 4096 (requested block: offset=217088, length=4096)
verify: bad header rand_seed 1674925967952414537, wanted 14984459301913261758 at file /export/core-file/fiotest123 offset 196608, length 4096 (requested block: offset=196608, length=4096)
verify: bad header rand_seed 4268553994388312182, wanted 457404114628905840 at file /export/core-file/fiotest123 offset 36864, length 4096 (requested block: offset=36864, length=4096)
verify: bad header rand_seed 16334828667629311866, wanted 2769163238992210515 at file /export/core-file/fiotest123 offset 65536, length 4096 (requested block: offset=65536, length=4096)
verify: bad header rand_seed 9149231600040020288, wanted 100525657264688800 at file /export/core-file/fiotest123 offset 241664, length 4096 (requested block: offset=241664, length=4096)
verify: bad header rand_seed 2539203786165567217, wanted 282301576315547703 at file /export/core-file/fiotest123 offset 258048, length 4096 (requested block: offset=258048, length=4096)
verify: bad header rand_seed 764909874126869656, wanted 8532681277354264188 at file /export/core-file/fiotest123 offset 90112, length 4096 (requested block: offset=90112, length=4096)
verify: bad header rand_seed 14786259703343837280, wanted 404540120011731088 at file /export/core-file/fiotest123 offset 122880, length 4096 (requested block: offset=122880, length=4096)
verify: bad header rand_seed 2386410779959611822, wanted 11074800191867490702 at file /export/core-file/fiotest123 offset 212992, length 4096 (requested block: offset=212992, length=4096)
fio: pid=494472, err=84/file:io_u.c:2190, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character

fiofiotest: (groupid=0, jobs=1): err=84 (file:io_u.c:2190, func=io_u_queued_complete, error=Invalid or incomplete multibyte or wide character): pid=494472: Tue Feb 14 13:04:34 2023
read: IOPS=38.3k, BW=149MiB/s (157MB/s)(4098MiB/27420msec)
slat (nsec): min=2000, max=186489, avg=2815.91, stdev=1167.94
clat (usec): min=19, max=5329, avg=874.89, stdev=213.83
lat (usec): min=21, max=5332, avg=877.71, stdev=213.84
clat percentiles (usec):
| 1.00th=[ 506], 5.00th=[ 701], 10.00th=[ 742], 20.00th=[ 783],
| 30.00th=[ 807], 40.00th=[ 832], 50.00th=[ 857], 60.00th=[ 881],
| 70.00th=[ 906], 80.00th=[ 938], 90.00th=[ 996], 95.00th=[ 1057],
| 99.00th=[ 1434], 99.50th=[ 1909], 99.90th=[ 3884], 99.95th=[ 4228],
| 99.99th=[ 4686]
bw ( KiB/s): min=128768, max=163256, per=100.00%, avg=153051.56, stdev=5859.65, samples=54
iops : min=32192, max=40814, avg=38262.89, stdev=1464.91, samples=54
write: IOPS=38.2k, BW=149MiB/s (157MB/s)(4094MiB/27419msec); 0 zone resets
slat (usec): min=16, max=256, avg=20.96, stdev= 2.96
clat (usec): min=13, max=1639, avg=753.77, stdev=105.51
lat (usec): min=33, max=1661, avg=774.73, stdev=105.70
clat percentiles (usec):
| 1.00th=[ 363], 5.00th=[ 603], 10.00th=[ 644], 20.00th=[ 693],
| 30.00th=[ 717], 40.00th=[ 742], 50.00th=[ 766], 60.00th=[ 783],
| 70.00th=[ 807], 80.00th=[ 832], 90.00th=[ 865], 95.00th=[ 889],
| 99.00th=[ 955], 99.50th=[ 988], 99.90th=[ 1156], 99.95th=[ 1303],
| 99.99th=[ 1483]
bw ( KiB/s): min=127776, max=164344, per=100.00%, avg=152930.07, stdev=6054.32, samples=54
iops : min=31944, max=41086, avg=38232.52, stdev=1513.58, samples=54
lat (usec) : 20=0.02%, 50=0.05%, 100=0.07%, 250=0.29%, 500=1.07%
lat (usec) : 750=26.38%, 1000=67.21%
lat (msec) : 2=4.68%, 4=0.20%, 10=0.04%
cpu : usr=73.65%, sys=26.14%, ctx=1794, majf=0, minf=24695
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=0.5%, >=64=98.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1049037,1048179,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=149MiB/s (157MB/s), 149MiB/s-149MiB/s (157MB/s-157MB/s), io=4098MiB (4297MB), run=27420-27420msec
WRITE: bw=149MiB/s (157MB/s), 149MiB/s-149MiB/s (157MB/s-157MB/s), io=4094MiB (4293MB), run=27419-27419msec

Disk stats (read/write):
nvme0n1: ios=1047861/1231648, merge=0/1, ticks=152484/22593, in_queue=175046, util=97.62%

xirs · 2023-02-14T06:44:39Z

but if use "-verify_async=4 --verify_backlog=8" parameter, it will run normally
fio-3.33 -name=fiofiotest -thread -group_reporting -ioengine=libaio -direct=1 -offset=0 -size=8G -numjobs=1 -bsrange=512-1M -iodepth=64 -rw=rw -verify=crc32 --verify_dump=1 --verify_fatal=1 -verify_async=4 --verify_backlog=8 -serialize_overlap=1 --eta=always -random_generator=tausworthe64 -filename=/export/core-file/fiotest123 -time_based -runtime=$((3600*24))

xirs · 2023-02-14T09:26:23Z

use below paramter，（-verify=md5, and without '-verify_async=4 --verify_backlog=8'), it still report error
[root@A06-R08-I132-3-614X44X fiotest]# fio-3.33 -name=fiofiotest -thread -group_reporting -ioengine=libaio -direct=1 -offset=0 -size=8G -numjobs=1 -bsrange=512-1M -iodepth=64 -rw=rw -verify=md5 --verify_dump=1 --verify_fatal=1 -serialize_overlap=1 --eta=always -filename=/export/core-file/fiotest123 -time_based -runtime=$((3600*24))
fiofiotest: (g=0): rw=rw, bs=(R) 512B-1024KiB, (W) 512B-1024KiB, (T) 512B-1024KiB, ioengine=libaio, iodepth=64
fio-3.33
Starting 1 thread
verify: bad header rand_seed 114498618377406544, wanted 7153490563776494 at file /export/core-file/fiotest123 offset 5467136, length 512 (requested block: offset=5467136, length=63488)
hdr_fail data dumped as fiotest123.5467136.hdr_fail
verify: bad header rand_seed 9852480210356360750, wanted 46386204153304124 at file /export/core-file/fiotest123 offset 0, length 512 (requested block: offset=0, length=148992)
hdr_fail data dumped as fiotest123.0.hdr_fail
verify: bad header rand_seed 1763778943062938676, wanted 9852480210356360750 at file /export/core-file/fiotest123 offset 148992, length 512 (requested block: offset=148992, length=691200)
hdr_fail data dumped as fiotest123.148992.hdr_fail
verify: bad header rand_seed 56763003395434021, wanted 4529082704144683152 at file /export/core-file/fiotest123 offset 2987008, length 512 (requested block: offset=2987008, length=126464)
hdr_fail data dumped as fiotest123.2987008.hdr_fail
verify: bad header rand_seed 12096295506076968138, wanted 1277040692297970740 at file /export/core-file/fiotest123 offset 4695040, length 512 (requested block: offset=4695040, length=260608)
hdr_fail data dumped as fiotest123.4695040.hdr_fail
verify: bad header rand_seed 9605910009376181991, wanted 456666279995977728 at file /export/core-file/fiotest123 offset 4277248, length 512 (requested block: offset=4277248, length=222208)
hdr_fail data dumped as fiotest123.4277248.hdr_fail
verify: bad header rand_seed 4392666214501339773, wanted 2869145916220254092 at file /export/core-file/fiotest123 offset 12825088, length 512 (requested block: offset=12825088, length=195584)
hdr_fail data dumped as fiotest123.12825088.hdr_fail
verify: bad header rand_seed 12376412139969171673, wanted 2618251529257221425 at file /export/core-file/fiotest123 offset 5530624, length 512 (requested block: offset=5530624, length=11776)
hdr_fail data dumped as fiotest123.5530624.hdr_fail
verify: bad header rand_seed 767616533229647040, wanted 948079092337337204 at file /export/core-file/fiotest123 offset 30187520, length 512 (requested block: offset=30187520, length=81920)

xirs · 2023-02-14T09:27:22Z

so the conclusion is

fio with '-verify_async=4 --verify_backlog=8' , won't report error
fio without '-verify_async=4 --verify_backlog=8' , will report error

horshack-dpreview · 2023-02-14T19:41:11Z

I've root-caused this issue. The random seed used to generate the data pattern for writes is getting out of sync with the reuse of that same seed for the verifies. Here's the flow:

When the job first starts the initial "verify seed" is randomly generated when init.c::add_job() calls setup_random_seeds()

fio/init.c

Lines 1236 to 1259 in 1bd16cf

    
           static int setup_random_seeds(struct thread_data *td) 
        
           { 
        
           	uint64_t seed; 
        
           	unsigned int i; 
        
           	if (!td->o.rand_repeatable && !fio_option_is_set(&td->o, rand_seed)) { 
        
           		int ret = init_random_seeds(td->rand_seeds, sizeof(td->rand_seeds)); 
        
           		if (!ret) 
        
           			td_fill_rand_seeds(td); 
        
           		return ret; 
        
           	} 
        
           	seed = td->o.rand_seed; 
        
           	for (i = 0; i < 4; i++) 
        
           		seed *= 0x9e370001UL; 
        
           	for (i = 0; i < FIO_RAND_NR_OFFS; i++) { 
        
           		td->rand_seeds[i] = seed * td->thread_number + i; 
        
           		seed *= 0x9e370001UL; 
        
           	} 
        
           	td_fill_rand_seeds(td); 
        
           	return 0; 
        
           }

Each Write I/O generated by backend.c::do_io() generates a random number from the "verify seed" to seed buffer fill with random data. The seed used is also stored in the header:

fio/backend.c

Lines 1035 to 1038 in 1bd16cf

    
           if (io_u->ddir == DDIR_WRITE && td->flags & TD_F_DO_VERIFY) { 
        
           	io_u->numberio = td->io_issues[io_u->ddir]; 
        
           	populate_verify_io_u(td, io_u); 
        
           }

When the workload completes its writes+reads, do_verify() is called to read-back the blocks written and verify the data. Before this begins the "verify seed" is reset back to where it was before the Write I/Os started, that way the same sequence of random numbers will be generated again, this time for verify reads. Each seed re-generated is compared against the seed in the header that's read back as part of the verify.

fio/backend.c

Lines 640 to 647 in 1bd16cf

    
           	/* 
        
           	 * verify_state needs to be reset before verification 
        
           	 * proceeds so that expected random seeds match actual 
        
           	 * random seeds in headers. The main loop will reset 
        
           	 * all random number generators if randrepeat is set. 
        
           	 */ 
        
           	if (!td->o.rand_repeatable) 
        
           		td_fill_verify_state_seed(td);

The first iteration through do_io() [performs workload writes+reads] and do_verify() [performs reads to verify data from workload's writes] complete without incident, successfully verifying the data.
Because the user specified time_based, more than one iteration of do_io() and do_verify() will execute. The same would also happen if loops= was specified. On the second iteration through do_io(), the "verify seed" is left in the state from the last completed verify, meaning it will generate unique "verify seeds" for the second set of writes. This is by design due to the user specifying randrepeat=0.
When the second iteration of do_io() completes and do_verify() is called, the "verify seed" is once again reset back to its initial job-start state. This means the same set of verify seeds will be generated as the first do_io() and do_verify() invocations, even though do_io() used a different seed starting point left over from the previous invocation. This causes the regenerated verify seeds to mismatch the verify seed written in the headers read by the verifies.

There is already logic in verify_io_u() to bypass checking of the "verify seed" for the case of write-only workloads, by forcing the seeds equal after the verify read but before the data is compared. Notice it also bypasses the check if verify_backlog is enabled, which is why the user's test doesn't fail when using the verify_backlog:

fio/verify.c

Lines 920 to 925 in 1bd16cf

    
           		 * Make rand_seed check pass when have verify_backlog or 
        
           		 * zone reset frequency for zonemode=zbd. 
        
           		 */ 
        
           		if (!td_rw(td) || (td->flags & TD_F_VER_BACKLOG) || 
        
           		    td->o.zrf.u.f) 
        
           			io_u->rand_seed = hdr->rand_seed;

Note td_rw() returns true if the workload has both writes+reads specified, ie rw=rw or rw=randrw.

There are multiple ways to fix this. One would be add write-only workloads as a case to ignore the random seed, although that weakens the integrity checking of the verify logic. Another way would be to regenerate the random seed for each iteration of do_io() for randrepeat=0. Yet another fix would be to store the "verify seed" in the io_piece structure that logs the writes - the logic in log_io_piece() already stores the numberio for a similar reason.

xirs · 2023-02-15T01:15:46Z

just wonder if fio with "-verify_async=4 --verify_backlog=8" (and with other parameters) can do verification rightly as expected?
if yes, I will use the parameter combination to do verify before this issue is fixed

horshack-dpreview · 2023-02-15T01:40:29Z

just wonder if fio with "-verify_async=4 --verify_backlog=8" (and with other parameters) can do verification rightly as expected? if yes, I will use the parameter combination to do verify before this issue is fixed

--verify_backlog=8 should be sufficient to avoid the issue; --verify_async shouldn't be necessary.

xirs · 2023-02-15T14:53:41Z

Thank you.
Wish this issue be fixed soon

Verify fails with "bad header rand_seed" when multiple iterations of do_io() execute (time_based=1 or loops>0), with verify enabled and randrepeat=0 The root cause is do_verify() resetting the verify seed back to the job-init value, which works for verification of the first iteration of do_io() but fails for subsequent iterations because the seed is left in its post-do_io() state after the first do_verify(), which means different rand values for the second iteration of do_io() yet the second iteration of do_verify() will revert back again to the job-init seed value. The fix is to revert the verify seed for randrepeat=0 back to ts state when do_io() last ran rather than to its job-init value. That will allow do_verify() to use the correct seed for each iteration while still retaining a per-iteration unique verify seed. Link: axboe#1517 (comment) Signed-off-by: Adam Horshack (horshack@live.com)

horshack-dpreview · 2023-02-23T00:37:59Z

@xirs, This fix for this issue has been merged. Can you please verify and close this issue? Thanks.

xirs · 2023-03-09T06:41:41Z

I have downloaded master branch and compiled fio binary, and will try it

xirs · 2023-03-10T06:28:08Z

I use below command, and it won't report error now.
fio-3.33 -name=fiofiotest -ioengine=libaio -direct=1 -verify=crc32 -thread -size=8G -offset=0 -time_based -numjobs=1 -bs=4K -iodepth=64 -rw=rw -randrepeat=0 -group_reporting -serialize_overlap 1 -filename=/export/core-file/fiotest123 -runtime=$((3600*24))

Verify fails with "bad header rand_seed" when multiple iterations of do_io() execute (time_based=1 or loops>0), with verify enabled and randrepeat=0 The root cause is do_verify() resetting the verify seed back to the job-init value, which works for verification of the first iteration of do_io() but fails for subsequent iterations because the seed is left in its post-do_io() state after the first do_verify(), which means different rand values for the second iteration of do_io() yet the second iteration of do_verify() will revert back again to the job-init seed value. The fix is to revert the verify seed for randrepeat=0 back to ts state when do_io() last ran rather than to its job-init value. That will allow do_verify() to use the correct seed for each iteration while still retaining a per-iteration unique verify seed. Link: axboe#1517 (comment) Signed-off-by: Adam Horshack (horshack@live.com)

horshack-dpreview mentioned this issue Feb 18, 2023

Bad header rand_seed with time_based or loops with randrepeat=0 verify #1522

Merged

xirs closed this as completed Mar 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fio 3.33 report Invalid or incomplete multibyte or wide character #1517

fio 3.33 report Invalid or incomplete multibyte or wide character #1517

xirs commented Feb 14, 2023 •

edited

Loading

xirs commented Feb 14, 2023

xirs commented Feb 14, 2023

xirs commented Feb 14, 2023

xirs commented Feb 14, 2023 •

edited

Loading

horshack-dpreview commented Feb 14, 2023 •

edited

Loading

xirs commented Feb 15, 2023

horshack-dpreview commented Feb 15, 2023 •

edited

Loading

xirs commented Feb 15, 2023

horshack-dpreview commented Feb 23, 2023

xirs commented Mar 9, 2023

xirs commented Mar 10, 2023 •

edited

Loading

fio 3.33 report Invalid or incomplete multibyte or wide character #1517

fio 3.33 report Invalid or incomplete multibyte or wide character #1517

Comments

xirs commented Feb 14, 2023 • edited Loading

xirs commented Feb 14, 2023

xirs commented Feb 14, 2023

xirs commented Feb 14, 2023

xirs commented Feb 14, 2023 • edited Loading

horshack-dpreview commented Feb 14, 2023 • edited Loading

xirs commented Feb 15, 2023

horshack-dpreview commented Feb 15, 2023 • edited Loading

xirs commented Feb 15, 2023

horshack-dpreview commented Feb 23, 2023

xirs commented Mar 9, 2023

xirs commented Mar 10, 2023 • edited Loading

xirs commented Feb 14, 2023 •

edited

Loading

xirs commented Feb 14, 2023 •

edited

Loading

horshack-dpreview commented Feb 14, 2023 •

edited

Loading

horshack-dpreview commented Feb 15, 2023 •

edited

Loading

xirs commented Mar 10, 2023 •

edited

Loading