filler log error: md5 x does not match x uploaded chunk s to the volume server

[aoberest]

Sponsors SeaweedFS via Patreon https://www.patreon.com/seaweedfs
Report issues here. Ask questions here https://stackoverflow.com/questions/tagged/seaweedfs
Please ask questions in https://github.com/chrislusf/seaweedfs/discussions

Hello,
I'm doing load testing using a project Minio Warp
In one of the tests, I started getting the filer error message:

Please see the log here

E0507 07:23:03 54425 filer_server_handlers_write_upload.go:78] md5 695ec683c26662c574dbaf9bfc33e5e0 does not match c5ff514ef36f6f0507214752310b9074 uploaded chunk 446.flO7F1vCcButnVHJ.rnd to the volume server
E0507 07:23:04 54425 filer_server_handlers_write_upload.go:78] md5 bfc1ca28b13c14e6660fee947aeb5631 does not match 27273b07b4e26c17130bf23e2300a441 uploaded chunk 463.IhnW8yHVFEeJ9mPP.rnd to the volume server
E0507 07:23:04 54425 filer_server_handlers_write_upload.go:78] md5 5d85060238a1be4c6b145d5099189ac1 does not match 89bb434c800f82be58b32ac76e47d4b7 uploaded chunk 420.U3sMMCKHDP(sFj8m.rnd to the volume server
E0507 07:23:04 54425 filer_server_handlers_write_upload.go:78] md5 6e3ecebdeb34bb0ea8ece82ff68e113c does not match 4d1cc93f793da35f07dc77bd12512560 uploaded chunk 434.DuLv5MsiTUWVBy9X.rnd to the volume server
E0507 07:23:04 54425 filer_server_handlers_write_upload.go:78] md5 fbf36700f92fc90f07962192c64a0217 does not match 87f11c0ebdf88e1cdc1c15efafa22a79 uploaded chunk 445.rp2i3F9xIu9dwsnS.rnd to the volume server
E0507 07:23:05 54425 filer_server_handlers_write_upload.go:78] md5 576c4f86f983b97a2af9b9a5f57bc76a does not match 12fb83cae15f83ff1b7377ea77558253 uploaded chunk 468.kstCxtaXn(2DozpT.rnd to the volume server
E0507 07:23:05 54425 filer_server_handlers_write_upload.go:78] md5 19dbc5746f850af980a135cfbdf9dc1d does not match a0f07de560cf1a49beea4a42d968b01c uploaded chunk 436.ImlvgPg2bKp1BzM(.rnd to the volume server
E0507 07:23:05 54425 filer_server_handlers_write_upload.go:78] md5 0b235b298ca08999c979acea54cce358 does not match d468155a8c028ebad9f13bd9965f9199 uploaded chunk 456.bcWPJaBaXF4prf)r.rnd to the volume server
E0507 07:23:06 54425 filer_server_handlers_write_upload.go:78] md5 53ce0270487571a5bff2a846248aa149 does not match 67c3d9302814d194856957098513a2ff uploaded chunk 465.ZuFkLQOMqc(RE0zi.rnd to the volume server
E0507 07:23:06 54425 filer_server_handlers_write_upload.go:78] md5 d1c2926f0a2904adbe1b20cb9a7085cc does not match 8bf9d14cf1f1ea1332c6eec82312f5f4 uploaded chunk 439.iyjbhHY5B5EtY1NU.rnd to the volume server
E0507 07:23:09 54425 filer_server_handlers_write_upload.go:78] md5 686544c385fcddfe296a3c98c9f57dd6 does not match f0a5a94025f5cd744d139968e2010d47 uploaded chunk 428.Khy1K33emqN2pDeP.rnd to the volume server
E0507 07:23:10 54425 filer_server_handlers_write_upload.go:78] md5 5c19d3bd1846359b42dd179179ab04a4 does not match 12670b25169d8213fe0b7e21a19c7eff uploaded chunk 464.IEg4wO7pNHG61zG).rnd to the volume server
E0507 07:23:10 54425 filer_server_handlers_write_upload.go:78] md5 5104a7f3e4ff3e28dbcfc37e365efed6 does not match 48341ee9951b65681f400e09000098db uploaded chunk 463.bz7g9SbLoC)PbZQ5.rnd to the volume server
E0507 07:23:11 54425 filer_server_handlers_write_upload.go:78] md5 0a6d0962172db5e92471db2dd5239c8d does not match 29f8b0f0c2b1bccce969d6055717c4ff uploaded chunk 454.NWVAbZLmV2CKfT2I.rnd to the volume server
E0507 07:23:11 54425 filer_server_handlers_write_upload.go:78] md5 38dd90b42fb5d69d5984c641711c10b5 does not match 74ae64b6fdc21480b4160b9019847331 uploaded chunk 451.NJ288cIypl(a1dSq.rnd to the volume server
E0507 07:23:11 54425 s3api_object_handlers.go:380] upload to filer error: read tcp 10.11.10.76:8888->10.11.10.76:35476: i/o timeout
E0507 07:23:12 54425 filer_server_handlers_write_upload.go:78] md5 7f0003f000e7943f5e5a3a13de51ace1 does not match 5dbdfa651b4b891912243fac9d8f965d uploaded chunk 440.2r)K1klG2Z4iDga7.rnd to the volume server
E0507 07:23:14 54425 filer_server_handlers_write_upload.go:78] md5 476145b0441535505a8865cb872edc42 does not match d3fa9637c75a810215d89075d78d24ef uploaded chunk 458.ai7kNcnBchKH2jQW.rnd to the volume server
E0507 07:23:14 54425 filer_server_handlers_write_upload.go:78] md5 424fe8ff3ec3e7e949a5e5a176f42940 does not match 2e6fb44ff54c75fc34add944321602d1 uploaded chunk 466.jnNoxdQ18W9FQaui.rnd to the volume server
E0507 07:23:14 54425 filer_server_handlers_write_upload.go:78] md5 2a07db3c0d068a08bbd9efd1573f9a4a does not match d50e80a5041d17e49043d2557a26e8ee uploaded chunk 431.j9(WkgdpIaImMJEc.rnd to the volume server
E0507 07:23:15 54425 filer_server_handlers_write_upload.go:78] md5 0b6f70160f62d819b6cefa3759d8aca7 does not match fa0e88bef5faeba5ad01504f631cf3cb uploaded chunk 450.rRTiNMscJxVtGNI5.rnd to the volume server
E0507 07:23:15 54425 filer_server_handlers_write_upload.go:78] md5 3da78afa9d130280d6c2aa8c47c6fd8d does not match 66d3fc0565b73a490685b1724a1fa3a6 uploaded chunk 447.pb8qfbJFM0n8SCh(.rnd to the volume server
E0507 07:23:15 54425 filer_server_handlers_write_upload.go:78] md5 576a8285a1b49e4d10839427d4751c4f does not match 769d28aacbe0ab24006a4e1905069f2f uploaded chunk 459.PicVruNVX8dlasup.rnd to the volume server
E0507 07:23:17 54425 filer_server_handlers_write_upload.go:78] md5 105de9814b43143c37e6885da43246d0 does not match 283fd7360e2a4a11e60f3a982378fa60 uploaded chunk 446.flO7F1vCcButnVHJ.rnd to the volume server
E0507 07:23:17 54425 filer_server_handlers_write_upload.go:78] md5 5da953876233d958bb8d91933326886d does not match e9efc85f2d61f9c674bbe2f2a3124a53 uploaded chunk 436.ImlvgPg2bKp1BzM(.rnd to the volume server
E0507 07:23:19 54425 filer_server_handlers_write_upload.go:78] md5 c3f3d966b593a2c15f47077feeffcd69 does not match 65de9062a4f3a53bed7aefa32367255e uploaded chunk 436.9YGFuGlJ1nM6OLnF.rnd to the volume server
E0507 07:23:19 54425 filer_server_handlers_write_upload.go:78] md5 7d0e9983c14239bd05cc76cd2a68d7db does not match 8d5f1bf35cc156e4786b2d0316b63ae6 uploaded chunk 454.Ljg3MpJe0ANSn)LF.rnd to the volume server
E0507 07:23:19 54425 filer_server_handlers_write_upload.go:78] md5 8bb98e9672f73c203274ad355cd86c4c does not match eec25ed9043a09685ad2e94c5e9b2e8d uploaded chunk 457.ex6oJwonutv43Hak.rnd to the volume server
E0507 07:23:21 54425 filer_server_handlers_write_upload.go:78] md5 2dab7c598d0c8d2604ff48fa943c5043 does not match e469b1d30fc251256936387972adb814 uploaded chunk 465.0EYfMip1rG7hWVNS.rnd to the volume server
E0507 07:23:21 54425 filer_server_handlers_write_upload.go:78] md5 274c4c6086eb9f1af5a6ecbb0afc93d6 does not match da6b553f87a62cce6e2b66c822a3b050 uploaded chunk 454.NWVAbZLmV2CKfT2I.rnd to the volume server
E0507 07:23:21 54425 filer_server_handlers_write_upload.go:78] md5 429797fba9a18d734430ac2193389688 does not match 4037f90a321bd9145d834b62cd00989c uploaded chunk 451.NJ288cIypl(a1dSq.rnd to the volume server
E0507 07:23:22 54425 filer_server_handlers_write_upload.go:78] md5 19f8f1e0d617aa421744f61ddde33edc does not match 4cff833da84b43bca5ecf971c4d9fb20 uploaded chunk 456.bcWPJaBaXF4prf)r.rnd to the volume server
E0507 07:23:22 54425 filer_server_handlers_write_upload.go:78] md5 eddbc35f090f5d31fac67efdea0f68b3 does not match d7f141c3e0c29e1a7c860982db482f39 uploaded chunk 466.jnNoxdQ18W9FQaui.rnd to the volume server
E0507 07:23:22 54425 filer_server_handlers_write_upload.go:78] md5 81943469b23f01dea18253892749807e does not match a5b4fbfcc3d43aef308c84742efb2a63 uploaded chunk 462.L(lM3NamX(4TGKem.rnd to the volume server
E0507 07:23:22 54425 filer_server_handlers_write_upload.go:78] md5 3f0e2d7c93e053379b2e3a3aeb5c2bfb does not match b5da49552751b8a71f18a7e4fd4da797 uploaded chunk 464.IEg4wO7pNHG61zG).rnd to the volume server
E0507 07:23:22 54425 filer_server_handlers_write_upload.go:78] md5 7623d79f01b85b33e8861d5915f22000 does not match 590b3ae0f9ffa38b367bb05627f64e8c uploaded chunk 465.ZuFkLQOMqc(RE0zi.rnd to the volume server
E0507 07:23:22 54425 filer_server_handlers_write_upload.go:78] md5 5f675f203f61a0d5ea7bfdbcd38dc506 does not match 61b67db98ae643edfd8ed42aae7ef5aa uploaded chunk 428.Khy1K33emqN2pDeP.rnd to the volume server
E0507 07:23:22 54425 filer_server_handlers_write_upload.go:78] md5 b28d584facd82efb66940d0c68327e07 does not match 6b69ec22e11677931ebdf639cb759d74 uploaded chunk 429.W6a0hFuTQy4hem)T.rnd to the volume server
E0507 07:23:22 54425 filer_server_handlers_write_upload.go:78] md5 c6edce9e554b5cc69d78202f9e8359e9 does not match f1ee765346dc229f8fb6255a7804d082 uploaded chunk 439.iyjbhHY5B5EtY1NU.rnd to the volume server
E0507 07:23:23 54425 filer_server_handlers_write_upload.go:78] md5 1a21250dcb0319d54dc943e875186528 does not match eecedc1323080504b9300dcf97e9edea uploaded chunk 465.0EYfMip1rG7hWVNS.rnd to the volume server
E0507 07:23:23 54425 filer_server_handlers_write_upload.go:78] md5 e92c7cebc5c8132e5c4ca460ec23a4a3 does not match 74b650bbda806001e7d2e2aec1dded26 uploaded chunk 434.g09OWM2(WXViTeQG.rnd to the volume server
E0507 07:23:24 54425 filer_server_handlers_write_upload.go:78] md5 33f43404a13e15cad2b69a826f67fd4f does not match 9ba30b23966178b6e1664a47f3d0e4c1 uploaded chunk 466.2AcfULwpUIdW5V1).rnd to the volume server
E0507 07:23:24 54425 filer_server_handlers_write_upload.go:78] md5 2bb37c216986665dab6127d0fe3c8672 does not match c3f918a135c900d1c7674ad8c245daf3 uploaded chunk 463.q21cyCDXLs7arZNC.rnd to the volume server
E0507 07:23:24 54425 filer_server_handlers_write_upload.go:78] md5 57f6dfb22a05be0e83d0629b4b3f7b6c does not match bc55c213301c064ea9323a6c9c70bcf0 uploaded chunk 452.L)4(xRzHaL(ikOGR.rnd to the volume server
E0507 07:23:24 54425 filer_server_handlers_write_upload.go:78] md5 b948109c09bb4401ea44fdf5bcaebed9 does not match b8a19e6e3dd7d9f090395cfac242fd00 uploaded chunk 460.v)hgRGvNAKyK2zfG.rnd to the volume server
E0507 07:23:25 54425 filer_server_handlers_write_upload.go:78] md5 728b1bae48b5162b44d96e4b56e22080 does not match 2097993ab2fca5a24ddd8dbc66671f48 uploaded chunk 426.3Jo(gxch5r5z6cTN.rnd to the volume server
E0507 07:23:25 54425 filer_server_handlers_write_upload.go:78] md5 92d764bf8646e7052fc942975696beb7 does not match c66c49df845fa9076ab8b86b19d90775 uploaded chunk 457.gSCcbUShguGjb4ZT.rnd to the volume server
E0507 07:23:25 54425 filer_server_handlers_write_upload.go:78] md5 c2d95980f3ae97897574846f6e40cd7b does not match 3516bbfb7f17bbd3e5653f75a790507f uploaded chunk 420.U3sMMCKHDP(sFj8m.rnd to the volume server
E0507 07:23:25 54425 filer_server_handlers_write_upload.go:78] md5 8e7f5d40f0f8b0e1ca3e2d64673ef168 does not match 569e5d3619bf17d283bc89ce76ccfd4e uploaded chunk 434.DuLv5MsiTUWVBy9X.rnd to the volume server
E0507 07:23:26 54425 filer_server_handlers_write_upload.go:78] md5 d43ee5b8b81b5c51283040563f12e32e does not match fe7f6c6d3b0aaf50e1eaa3ac8aa1c457 uploaded chunk 464.V0VSIHMfAq)VUT8t.rnd to the volume server
E0507 07:23:28 54425 filer_server_handlers_write_upload.go:78] md5 9af25af35f929294141fb58085b6199d does not match 1d398fa73a7fddc32dc03e8d779057d6 uploaded chunk 473.3pfu)xAeIq51TCdq.rnd to the volume server
E0507 07:23:28 54425 filer_server_handlers_write_upload.go:78] md5 0dd388ff64a4f9e563945f955a30bc82 does not match 66a9099cfbcda4a39fe14d4a22a67ce3 uploaded chunk 459.VcbVa2YbVtJMElcG.rnd to the volume server
E0507 07:23:29 54425 filer_server_handlers_write_upload.go:78] md5 ca8311755ff6a712d4d69fef19fa72a0 does not match f5a3b602a69a219e91ded8b78cf5efad uploaded chunk 431.chbaLIvzfbNl4tyM.rnd to the volume server
E0507 07:23:29 54425 filer_server_handlers_write_upload.go:78] md5 51865b7319d0b7e7285250e94bebeea3 does not match cdba54e3f53310ec2eb5a27163c1b84a uploaded chunk 463.q21cyCDXLs7arZNC.rnd to the volume server
E0507 07:23:29 54425 filer_server_handlers_write_upload.go:78] md5 13fae3347d01e51df6bd746aed90ee95 does not match a225ccfde904ccc186369d28a2813819 uploaded chunk 447.pb8qfbJFM0n8SCh(.rnd to the volume server
E0507 07:23:30 54425 filer_server_handlers_write_upload.go:78] md5 53b94142d5491fcbf74ee2f432cc2084 does not match bb386f08ab41a9864df75358db35c827 uploaded chunk 462.L(lM3NamX(4TGKem.rnd to the volume server
E0507 07:23:31 54425 filer_server_handlers_write_upload.go:78] md5 3d51b8866496aa4e42129dc3d800da7f does not match 2325ddb595a44d8e2715083f844cf5c0 uploaded chunk 463.hSaCveMHpXX6YSw0.rnd to the volume server
E0507 07:23:31 54425 filer_server_handlers_write_upload.go:78] md5 5ec8c0e09657746bd9ecae9ef6f1bc20 does not match deced60d6a7517fc2a500dd744fd802a uploaded chunk 468.RaLOkFRLjxOBdvYw.rnd to the volume server
E0507 07:23:32 54425 filer_server_handlers_write_upload.go:78] md5 033d43db0b6751da54887bbbb1b52898 does not match 321cb1ad43b012c6da5e4320bc0a162e uploaded chunk 466.f(LTVcC5n0Mdn9TH.rnd to the volume server
E0507 07:23:32 54425 filer_server_handlers_write_upload.go:78] md5 a3d292edd85df6f3b57de23e5ab5a397 does not match 6f18d1902586c018a0314299917dd004 uploaded chunk 468.zyp2DUUENIl0h9)s.rnd to the volume server
E0507 07:23:32 54425 filer_server_handlers_write_upload.go:78] md5 f53a2cad97b31daf87a9e18dc1cb40b4 does not match f0af6b6bc3887fc1bfec10f392922755 uploaded chunk 458.ai7kNcnBchKH2jQW.rnd to the volume server
E0507 07:23:33 54425 filer_server_handlers_write_upload.go:78] md5 7a6a405053b4794b34cb38e8ee23f5c3 does not match 42fc1fffab0ccab4b63e98ccb8475a9c uploaded chunk 457.ex6oJwonutv43Hak.rnd to the volume server
E0507 07:23:33 54425 filer_server_handlers_write_upload.go:78] md5 9ca88c8dc79e2288c1d3222623d468bb does not match 245580fd8d6566e11e9440fa8ee977b9 uploaded chunk 425.ESBrD)AVrTEjn6DK.rnd to the volume server
E0507 07:23:34 54425 filer_server_handlers_write_upload.go:78] md5 10b36a55950bef9fa830b6381ecfa217 does not match 87f9edb74c1a5340a8dd12309f7ced62 uploaded chunk 428.sPN7VR3H76hwNw64.rnd to the volume server
E0507 07:23:35 54425 filer_server_handlers_write_upload.go:78] md5 70af5f91459ce320ebf87382da8e1e86 does not match 19aecaa7891008811fe2b9ab06fa199b uploaded chunk 451.FDFIbXakCUhGX8Ry.rnd to the volume server
E0507 07:23:36 54425 filer_server_handlers_write_upload.go:78] md5 b6ad3f6b2057f9a5a7d5eebe1f7c40c2 does not match f7089925ad74e3f9e4cc58ec9caca5c5 uploaded chunk 436.vOC4j3vAWhywPEgM.rnd to the volume server
E0507 07:23:36 54425 filer_server_handlers_write_upload.go:78] md5 0b4319a8787f8cc186a042beff8ff27b does not match e070939480cb35c1e2ae72903194bdc2 uploaded chunk 468.RaLOkFRLjxOBdvYw.rnd to the volume server
E0507 07:23:37 54425 filer_server_handlers_write_upload.go:78] md5 75a4bd35bdc86829324232bfec9643c2 does not match d7479b576342ff33a65201286dd42586 uploaded chunk 460.v)hgRGvNAKyK2zfG.rnd to the volume server
E0507 07:23:39 54425 filer_server_handlers_write_upload.go:78] md5 2836f09d9c7fa7a7d08574cd3d583c6e does not match a9dcb9471bb4ba98213f1124c9ec17f7 uploaded chunk 459.VcbVa2YbVtJMElcG.rnd to the volume server
E0507 07:23:39 54425 filer_server_handlers_write_upload.go:78] md5 2adc18bf0f82ebd96b1ba9ceb11ea2e8 does not match 2a5c7831a48acd8b2cdcec740eff63e9 uploaded chunk 464.V0VSIHMfAq)VUT8t.rnd to the volume server
E0507 07:23:39 54425 filer_server_handlers_write_upload.go:78] md5 00db85938f9a31ee639f499e5a4787d4 does not match 32b6c2a03170fa1f7d817bd7123477a3 uploaded chunk 466.2AcfULwpUIdW5V1).rnd to the volume server
E0507 07:23:39 54425 filer_server_handlers_write_upload.go:78] md5 83da8952eec894e33172878a4a801b8e does not match ce3f9e24d58d255cf19e138c8b741ea8 uploaded chunk 431.CYlaqnJiA46BDmoc.rnd to the volume server
E0507 07:23:40 54425 filer_server_handlers_write_upload.go:78] md5 d7e5b8f883967b372b010aabe3bcbbc9 does not match 1e7e0a7bf99fae89c64542ccf3ca0392 uploaded chunk 457.gSCcbUShguGjb4ZT.rnd to the volume server
E0507 07:23:40 54425 filer_server_handlers_write_upload.go:78] md5 942ab27f0aa470292ff2c5094ac98e48 does not match 32671ef896f7b2f7f79dfad21b317f17 uploaded chunk 473.3pfu)xAeIq51TCdq.rnd to the volume server
E0507 07:23:41 54425 filer_server_handlers_write_upload.go:78] md5 13a5a38f0fcde1a59e3b10f6e0924815 does not match 5dfd591e5e00badf0c96e56aaa94b808 uploaded chunk 452.L)4(xRzHaL(ikOGR.rnd to the volume server
E0507 07:23:41 54425 filer_server_handlers_write_upload.go:78] md5 bfad1ce980ac902bcb256d3c23ff1ca8 does not match fe19d22c847ef0b673c133125a4f65c2 uploaded chunk 434.g09OWM2(WXViTeQG.rnd to the volume server
E0507 07:23:42 54425 filer_server_handlers_write_upload.go:78] md5 eea8c75df6def44e3a8c87a90ec95cf0 does not match b8d146ec7278e265da8078e8368a80f2 uploaded chunk 468.zyp2DUUENIl0h9)s.rnd to the volume server
E0507 07:23:43 54425 filer_server_handlers_write_upload.go:78] md5 128b3f48f8512318b2472e720b5c80eb does not match 97a32698e981198c102593a2c4426fe3 uploaded chunk 426.cf7yPJ0sWvcFzrUU.rnd to the volume server
E0507 07:23:43 54425 filer_server_handlers_write_upload.go:78] md5 30c129c9cae05cd00307b3f3c49efffa does not match 6a1380f276bdcec0b6f9aba107051863 uploaded chunk 476.o53sv84VQtd0Wzh).rnd to the volume server
E0507 07:23:43 54425 filer_server_handlers_write_upload.go:78] md5 946506cd2af52132629cc1af857b8567 does not match 22d6e9e50d366022dcc7ff0196910a7a uploaded chunk 423.vMcd)tmcUcElWpob.rnd to the volume server
E0507 07:23:43 54425 filer_server_handlers_write_upload.go:78] md5 f8c06af8ef4dbb585d1f693d03bdeb42 does not match b2369417ca99c9fd73de607dea52a486 uploaded chunk 464.hUoy(QOX(HKhU)sd.rnd to the volume server
E0507 07:23:44 54425 filer_server_handlers_write_upload.go:78] md5 8e4d90f332ec02059286c4ea2f270f9c does not match 1e444437236fa55b915d6fb249d3704a uploaded chunk 457.0zYpDgQV5yQw6IW(.rnd to the volume server
E0507 07:23:45 54425 filer_server_handlers_write_upload.go:78] md5 52fc3dfe5c0ed04e09bd4535c69b11e1 does not match 731925fa3facbea61d702ede8ad73224 uploaded chunk 436.vOC4j3vAWhywPEgM.rnd to the volume server
E0507 07:23:45 54425 filer_server_handlers_write_upload.go:78] md5 f39bd4af12287aafdf6ffb0664718c19 does not match 4f0c550f6f2a58d041a471e809a2d553 uploaded chunk 429.W6a0hFuTQy4hem)T.rnd to the volume server
E0507 07:23:46 54425 filer_server_handlers_write_upload.go:78] md5 3a13f72c6cd7283bb347e3913c0f0c59 does not match e2d7d1b46ef17fd4b98a15a686e6c278 uploaded chunk 425.ESBrD)AVrTEjn6DK.rnd to the volume server
E0507 07:23:46 54425 filer_server_handlers_write_upload.go:78] md5 b218cb74e42e7e6a1c6e25aabd1fc9e7 does not match 86697f78acfceba4fbe46d5765cd4ac9 uploaded chunk 469.ZtbQecc83E)D09FD.rnd to the volume server
E0507 07:23:46 54425 filer_server_handlers_write_upload.go:78] md5 d887c43268f958a78bb9e19f7c4eeed1 does not match a758d4d8991c353c9df19853f56dc848 uploaded chunk 471.YC)QLLToujpfTpy0.rnd to the volume server
E0507 07:23:47 54425 filer_server_handlers_write_upload.go:78] md5 f5645931d071868f0e5461290dbf54c7 does not match dbabd3a6c2e0fb93fd70bb5f6863fd6a uploaded chunk 469.b0fIBK4u(wyGy7Kg.rnd to the volume server
E0507 07:23:48 54425 filer_server_handlers_write_upload.go:78] md5 9d2d76145fdc6345c44f6a1a05ac0898 does not match 061647b45f5119472357ef8d6cfec1f5 uploaded chunk 451.FDFIbXakCUhGX8Ry.rnd to the volume server
E0507 07:23:49 54425 filer_server_handlers_write_upload.go:78] md5 df5c7cb2d5fb44584bc87dc3ab379b6d does not match 05dbd11440053f900709e0705fd73feb uploaded chunk 467.TTHzlAnsevw7z(ON.rnd to the volume server
E0507 07:23:49 54425 filer_server_handlers_write_upload.go:78] md5 98ac63b4c0067e8f84f2abd1be474eaa does not match c385d847ca6ba6698182c6d9cb764cba uploaded chunk 471.YC)QLLToujpfTpy0.rnd to the volume server
E0507 07:23:49 54425 filer_server_handlers_write_upload.go:78] md5 0b814d060b8dee62ea9325f82532fb6a does not match 4fd8315df665d0b163797d7b265d2e49 uploaded chunk 463.Ntvi7mxfNJQVKw4w.rnd to the volume server
E0507 07:23:50 54425 filer_server_handlers_write_upload.go:78] md5 860ad1526fa9e893ec81e70272ad3597 does not match be18b5f96d9a833b528604025b7bc056 uploaded chunk 465.LqHILHnYYFjLfgI7.rnd to the volume server
E0507 07:23:50 54425 filer_server_handlers_write_upload.go:78] md5 d81b982df9b1d24c4e6217134d555a00 does not match 59da32ffd300ee314adf63f2f96ad018 uploaded chunk 466.xBe8mvs84BMLYZkE.rnd to the volume server
E0507 07:23:50 54425 filer_server_handlers_write_upload.go:78] md5 8b70a1a16c20b88459e666428a6bddef does not match d58bdb2946f1ad12d6d8c334bfa3d316 uploaded chunk 464.hUoy(QOX(HKhU)sd.rnd to the volume server
E0507 07:23:50 54425 filer_server_handlers_write_upload.go:78] md5 36956b959457f38dd2721cf6d0d18ec6 does not match 783df8d3198b5d06bff17d7a1c2f5e5a uploaded chunk 476.o53sv84VQtd0Wzh).rnd to the volume server
E0507 07:23:50 54425 filer_server_handlers_write_upload.go:78] md5 124f3bbb679cd0c7dbdbbce68126a742 does not match 6369a20a546a47903afd1f61b81edec7 uploaded chunk 463.KUTCuIeO)JybYu2Y.rnd to the volume server
E0507 07:23:51 54425 filer_server_handlers_write_upload.go:78] md5 96a0fafcb1737f2b06ec6d4b3459370b does not match ed431b57ffacc9cec1cbd3469d0a3a65 uploaded chunk 463.MO0)V2zUsABKmR7V.rnd to the volume server
E0507 07:23:51 54425 filer_server_handlers_write_upload.go:78] md5 c0f325bb84463e29ceb5b5e725bc340f does not match 490295199a48d768e431269ec0edce37 uploaded chunk 469.ZtbQecc83E)D09FD.rnd to the volume server
E0507 07:23:51 54425 filer_server_handlers_write_upload.go:78] md5 a83aeefd2dda8873f449a813473da098 does not match 397eade5acc1be2ec70daf3551503242 uploaded chunk 466.f(LTVcC5n0Mdn9TH.rnd to the volume server

My configuration.

defaultreplication  = '101'
masters   = 10.10.10.145:9333,10.10.10.146:9333,10.10.10.148:9333,10.11.10.72:9333,10.11.10.73:9333,10.11.10.74:9333,10.12.10.20:9333
volumes   = 10.10.10.145:8080,10.10.10.146:8080,10.10.10.148:8080,10.10.10.150:8080,10.11.10.72:8080,10.11.10.73:8080,10.11.10.74:8080,10.11.10.75:8080,10.11.10.76:8080,10.11.10.77:8080
filers    = 10.11.10.76:8888,10.10.10.148:8888,10.11.10.75:8888,10.10.10.150:8888
filers have peers = 10.11.10.76:8888,10.10.10.148:8888,10.11.10.75:8888,10.10.10.150:8888

filer1 - 10.10.10.148, 10.10.10.150 - posgresql 9.6 store
filer2 - 10.11.10.76, 10.11.10.75  - other posgresql 9.6 store

• OS version - CentOS Linux release 7.6.1810
• version 30GB 2.43 c48ef78 linux amd64

Filer1 filer.toml

# A sample TOML config file for SeaweedFS filer store
# Used with "weed filer" or "weed server -filer"
# Put this file to one of the location, with descending priority
#    ./filer.toml
#    $HOME/.seaweedfs/filer.toml
#    /etc/seaweedfs/filer.toml

####################################################
# Customizable filer server options
####################################################
[filer.options]
# with http DELETE, by default the filer would check whether a folder is empty.
# recursive_delete will delete all sub folders and files, similar to "rm -Rf"
recursive_delete = false
# directories under this folder will be automatically creating a separate bucket
buckets_folder = "/buckets"

####################################################
# The following are filer store options
####################################################



[postgres2]
enabled = true
createTable = """
  CREATE TABLE IF NOT EXISTS "%s" (
    dirhash   BIGINT, 
    name      VARCHAR(65535), 
    directory VARCHAR(65535), 
    meta      bytea, 
    PRIMARY KEY (dirhash, name)
  );
"""
hostname = "10.11.10.253"
port = 5432
username = "seaweedfs"
password = "pass"
database = "seaweedfs"          # create or use an existing database
schema = ""
sslmode = "disable"
connection_max_idle = 100
connection_max_open = 100
connection_max_lifetime_seconds = 0
# if insert/upsert failing, you can disable upsert or update query syntax to match your RDBMS syntax:
enableUpsert = true
upsertQuery = """INSERT INTO "%[1]s" (dirhash,name,directory,meta) VALUES($1,$2,$3,$4) ON CONFLICT (dirhash,name) DO UPDATE SET meta = EXCLUDED.meta WHERE "%[1]s".meta != EXCLUDED.meta"""

Expected behavior
Working without errors

Cluster status
Volume Size Limit 1024MB
Free 1782
Max 3501
Leader 10.11.10.73:9333
System Stats
Concurrent Connections 60
Up Time 65h0m56.14162639s

Why do I get these errors?
How to respond to them?
How do I fix the error or maybe I should change the cluster configuration?

[aoberest]

Warp config

--duration=640m --concurrent 40 --debug

Mixed operations.
Operation: DELETE, 10%, Concurrency: 80, Duration: 10h39m59s.
Errors: 14
* Throughput: 0.69 obj/s

Operation: GET, 45%, Concurrency: 80, Duration: 10h40m3s.
Errors: 84
* Throughput: 30.87 MiB/s, 3.09 obj/s

Operation: PUT, 15%, Concurrency: 80, Duration: 10h40m8s.
Errors: 173
* Throughput: 10.24 MiB/s, 1.02 obj/s

Operation: STAT, 30%, Concurrency: 80, Duration: 10h40m0s.
Errors: 43
* Throughput: 2.06 obj/s

Cluster Total: 41.11 MiB/s, 6.86 obj/s, 314 errors over 10h40m0s.
Total Errors:314.

[chrislusf]

This error message was an incorrect check and has been fixed already in the master branch.