[Critical][volume] many broken volumes

[kmlebedev]

Describe the bug
log

E0317 09:01:51     1 stream.go:39] read chunk: http://fast-volume-2.dc1:8080/367,891da0decf4e70?readDeleted=true: 404
E0317 09:01:51     1 stream.go:39] read chunk: http://fast-volume-2.dc1:8080/367,891da0decf4e70?readDeleted=true: 404
2021/03/17 09:01:51 http: superfluous response.WriteHeader call from github.com/chrislusf/seaweedfs/weed/server.processRangeRequest (common.go:284)

on fast-volume-4.dc2 content not exist

curl -v "http://fast-volume-2.dc1:8080/367,891da0decf4e70?readDeleted=true"
> GET /367,891da0decf4e70?readDeleted=true HTTP/1.1
> User-Agent: curl/7.29.0
> Host: fast-volume-2.dc1:8080
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: SeaweedFS Volume 30GB 2.26
< Date: Thu, 18 Mar 2021 11:00:43 GMT
< Content-Length: 0
<

on fast-volume-4.dc2 content exist

curl -v "http://fast-volume-4.dc2:8080/367,891da0decf4e70?readDeleted=true"
> GET /367,891da0decf4e70?readDeleted=true HTTP/1.1
> User-Agent: curl/7.29.0
> Host: fast-volume-4.dc2:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 71
< Content-Type: text/plain; charset=utf-8
< Etag: "b1a2b2cb"
< Last-Modified: Thu, 11 Mar 2021 10:12:57 GMT
< Server: SeaweedFS Volume 30GB 2.26
< Date: Thu, 18 Mar 2021 10:59:08 GMT
<

volume.list

      DataNode fast-volume-2.dc1:8080 volume:97/150 active:97 free:53 remote:0
        volume id:367  size:6140577136  collection:"registry"  file_count:3013  delete_count:316  deleted_byte_count:476614747  replica_placement:100  version:3  compact_revision:4  modified_at_second:1616063816
      DataNode fast-volume-4.dc2:8080 volume:73/120 active:73 free:47 remote:0
        volume id:367  size:6173020752  collection:"registry"  file_count:3032  delete_count:313  deleted_byte_count:476614866  replica_placement:100  version:3  compact_revision:4  modified_at_second:1616063816

fs.meta.cat

{
  "name": "link",
  "isDirectory": false,
  "chunks": [
    {
      "fileId": "367,891da2222c77d3",
      "offset": "0",
      "size": "71",
      "mtime": "1615457578510650003",
      "eTag": "8463baa673a77a7ab245534ba154401d",
      "sourceFileId": "",
      "fid": {
        "volumeId": 367,
        "fileKey": "8986018",
        "cookie": 573339603
      },
      "sourceFid": null,
      "cipherKey": null,
      "isCompressed": true,
      "isChunkManifest": false
    }
  ],
  "attributes": {
    "fileSize": "71",
    "mtime": "1615457578",
    "fileMode": 432,
    "uid": 0,
    "gid": 0,
    "crtime": "1615457577",
    "mime": "",
    "replication": "100",
    "collection": "registry",
    "ttlSec": 0,
    "userName": "",
    "groupName": [
    ],
    "symlinkTarget": "",
    "md5": "hGO6pnOnenqyRVNLoVRAHQ==",
    "diskType": ""
  },
  "extended": {
    "x-amz-storage-class": "U1RBTkRBUkQ="
  },
  "hardLinkId": null,
  "hardLinkCounter": 0,
  "content": null
}

see_idx
on volume-2 not found

./see_idx -collection registry -dir /tmp/volume-2 -volumeId 367 | grep 891da0

on volume-4 found

./see_idx -collection registry -dir /tmp/volume-4 -volumeId 367 | grep 891da0
key:891da0 offset:631678546 size:119(119 B)

diff

diff /tmp/volume-2/registry_367.see /tmp/volume-4/registry_367.see
1807d1806
< key:7be459
1846d1844
< key:7d6964
1950d1947
< key:80fdee
1978d1974
< key:824e12
2406d2401
< key:88c49a
2501d2495
< key:8913b6
2578d2571
< key:891d52
2584a2578,2599
> key:891d80
> key:891d82
> key:891d84
> key:891d86
> key:891d96
> key:891d98
> key:891d9a
> key:891d9e
> key:891da0
> key:891da2
> key:891da4
> key:891da6
> key:891dd7
> key:891ddd
> key:891dde
> key:891dfe
> key:891e06
> key:891e24
> key:891e26
> key:891e30
> key:891e50
> key:891e54
2736a2752
> key:891da6
2964d2979
< key:8c228f
2965a2981
> key:8c228f

System Setup
SeaweedFS Volume 30GB 2.26

Expected behavior

volumes do not break

[chrislusf]

related to #1519

[kmlebedev]

related to #1519

@chrislusf
This is good for solving the consequence of the problem, but long and expensive.

There may be a problem in the delay in rebuilding the topology of volumes, such operations as adding, removing and setting read-only attributes ?

It also seems to have a log of all operations from where the volume can get N last operations and check with what it has ?

Perhaps, as a temporary and quick solution, it is worth adding a shell command that will fix this?

Any other ideas?

[chrislusf]

we can take this approach #1777

Use the admin scripts to find and fix mismatches.