Non-replicated volume gets removed because of over replication

[divanikus]

I'm running a cluster with 000 replication and two tiers (ssd and hdd). Some volumes are removed after moving between tiers for some reason yet unknown.

Feb 02 08:31:53 wm-mb-00 weed[19634]: I0202 08:31:53 19634 volume_layout.go:386] Volume 1137830 becomes writable
Feb 02 08:31:53 wm-mb-00 weed[19634]: I0202 08:31:53 19634 volume_growth.go:235] Created Volume 1137830 on topo:dc1:hs00:192.168.65.66:8080
Feb 02 08:40:29 wm-mb-00 weed[19634]: I0202 08:40:29 19634 volume_layout.go:449] Volume 1137830 becomes crowded
Feb 02 08:41:56 wm-mb-00 weed[19634]: I0202 08:41:56 19634 volume_layout.go:373] Volume 1137830 becomes unwritable
Feb 03 02:59:21 wm-mb-00 weed[19634]: tier move volumes: [1137807 1137786 1137819 1137837 1137791 1137821 1137817 1137805 1137813 1137804 1137798 1137827 1137826 1137799 1137820 1137835 1137825 1137824 1137831 1137829 1137834 1137836 1137810 1137811 1137800 1137833 1137796 1137816 1137822 1137802 1137832 1137815 1137830 1137806]
Feb 03 03:49:50 wm-mb-00 weed[19634]: moving volume 1137830 from 192.168.65.66:8080 to 192.168.65.51:8080 with disk type hdd ...
Feb 03 03:49:50 wm-mb-00 weed[19634]: markVolumeReadonly 1137830 on 192.168.65.66:8080 ...
Feb 03 03:49:50 wm-mb-00 weed[19634]: 2022/02/03 03:49:50 copying volume 1137830 from 192.168.65.66:8080 to 192.168.65.51:8080
Feb 03 03:49:51 wm-mb-00 weed[19634]: volume 1137830 processed 136314880 bytes
Feb 03 03:49:53 wm-mb-00 weed[19634]: volume 1137830 processed 272629760 bytes
Feb 03 03:49:54 wm-mb-00 weed[19634]: volume 1137830 processed 408944640 bytes
Feb 03 03:49:55 wm-mb-00 weed[19634]: volume 1137830 processed 545259520 bytes
Feb 03 03:49:56 wm-mb-00 weed[19634]: volume 1137830 processed 681574400 bytes
Feb 03 03:49:57 wm-mb-00 weed[19634]: volume 1137830 processed 817889280 bytes
Feb 03 03:49:59 wm-mb-00 weed[19634]: volume 1137830 processed 954204160 bytes
Feb 03 03:50:00 wm-mb-00 weed[19634]: volume 1137830 processed 1090519040 bytes
Feb 03 03:50:01 wm-mb-00 weed[19634]: volume 1137830 processed 1226833920 bytes
Feb 03 03:50:02 wm-mb-00 weed[19634]: volume 1137830 processed 1363148800 bytes
Feb 03 03:50:03 wm-mb-00 weed[19634]: volume 1137830 processed 1499463680 bytes
Feb 03 03:50:04 wm-mb-00 weed[19634]: volume 1137830 processed 1635778560 bytes
Feb 03 03:50:06 wm-mb-00 weed[19634]: volume 1137830 processed 1772093440 bytes
Feb 03 03:50:07 wm-mb-00 weed[19634]: volume 1137830 processed 1908408320 bytes
Feb 03 03:50:08 wm-mb-00 weed[19634]: volume 1137830 processed 2044723200 bytes
Feb 03 03:50:09 wm-mb-00 weed[19634]: volume 1137830 processed 2181038080 bytes
Feb 03 03:50:10 wm-mb-00 weed[19634]: volume 1137830 processed 2317352960 bytes
Feb 03 03:50:12 wm-mb-00 weed[19634]: volume 1137830 processed 2453667840 bytes
Feb 03 03:50:13 wm-mb-00 weed[19634]: volume 1137830 processed 2589982720 bytes
Feb 03 03:50:15 wm-mb-00 weed[19634]: volume 1137830 processed 2726297600 bytes
Feb 03 03:50:17 wm-mb-00 weed[19634]: volume 1137830 processed 2862612480 bytes
Feb 03 03:50:20 wm-mb-00 weed[19634]: volume 1137830 processed 2998927360 bytes
Feb 03 03:50:23 wm-mb-00 weed[19634]: volume 1137830 processed 3135242240 bytes
Feb 03 03:51:53 wm-mb-00 weed[19634]: volume 1137830 processed 3271557120 bytes
Feb 03 03:51:54 wm-mb-00 weed[19634]: volume 1137830 processed 3407872000 bytes
Feb 03 03:51:55 wm-mb-00 weed[19634]: volume 1137830 processed 3544186880 bytes
Feb 03 03:51:57 wm-mb-00 weed[19634]: volume 1137830 processed 3680501760 bytes
Feb 03 03:51:58 wm-mb-00 weed[19634]: volume 1137830 processed 3816816640 bytes
Feb 03 03:51:59 wm-mb-00 weed[19634]: volume 1137830 processed 3953131520 bytes
Feb 03 03:52:00 wm-mb-00 weed[19634]: volume 1137830 processed 4089446400 bytes
Feb 03 03:52:01 wm-mb-00 weed[19634]: volume 1137830 processed 4225761280 bytes
Feb 03 03:52:03 wm-mb-00 weed[19634]: volume 1137830 processed 4362076160 bytes
Feb 03 03:52:04 wm-mb-00 weed[19634]: volume 1137830 processed 4498391040 bytes
Feb 03 03:52:05 wm-mb-00 weed[19634]: volume 1137830 processed 4634705920 bytes
Feb 03 03:52:06 wm-mb-00 weed[19634]: volume 1137830 processed 4771020800 bytes
Feb 03 03:52:07 wm-mb-00 weed[19634]: volume 1137830 processed 4907335680 bytes
Feb 03 03:52:09 wm-mb-00 weed[19634]: volume 1137830 processed 5043650560 bytes
Feb 03 03:52:10 wm-mb-00 weed[19634]: volume 1137830 processed 5179965440 bytes
Feb 03 03:52:11 wm-mb-00 weed[19634]: 2022/02/03 03:52:11 tailing volume 1137830 from 192.168.65.66:8080 to 192.168.65.51:8080
Feb 03 03:52:11 wm-mb-00 weed[19634]: I0203 03:52:11 19634 volume_layout.go:386] Volume 1137830 becomes writable
Feb 03 03:52:23 wm-mb-00 weed[19634]: 2022/02/03 03:52:23 deleting volume 1137830 from 192.168.65.66:8080
Feb 03 03:52:23 wm-mb-00 weed[19634]: 2022/02/03 03:52:23 moved volume 1137830 from 192.168.65.66:8080 to 192.168.65.51:8080
Feb 03 03:52:23 wm-mb-00 weed[19634]: I0203 03:52:23 19634 topology.go:209] removing volume info: Id:1137830, Size:0, ReplicaPlacement:000, Collection:mb-steps-2022-02-02-08, Version:3, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false from 192.168.65.66:8080
Feb 03 03:52:23 wm-mb-00 weed[19634]: volume 1137830 replication 000, but over replicated +2
Feb 03 03:52:23 wm-mb-00 weed[19634]: deleting volume 1137830 from 192.168.65.51:8080 ...
Feb 03 03:54:35 wm-mb-00 weed[19634]: I0203 03:54:35 19634 volume_layout.go:373] Volume 1137830 becomes unwritable
Feb 03 03:55:52 wm-mb-00 weed[19634]: I0203 03:55:52 19634 data_node.go:78] Deleting volume id: 1137830
Feb 03 03:55:52 wm-mb-00 weed[19634]: I0203 03:55:52 19634 topology.go:209] removing volume info: Id:1137830, Size:5283003224, ReplicaPlacement:000, Collection:mb-steps-2022-02-02-08, Version:3, FileCount:176442, DeleteCount:207, DeletedByteCount:5645668, ReadOnly:false from 192.168.65.51:8080
Feb 03 03:55:52 wm-mb-00 weed[19634]: I0203 03:55:52 19634 master_grpc_server.go:125] master see deleted volume 1137830 from 192.168.65.51:8080
Feb 03 03:55:52 wm-mb-00 weed[19634]: I0203 03:55:52 19634 topology.go:209] removing volume info: Id:1137830, Size:0, ReplicaPlacement:000, Collection:mb-steps-2022-02-02-08, Version:3, FileCount:0, DeleteCount:0, DeletedByteCount:0, ReadOnly:false from 192.168.65.51:8080

I might be thinking that latest parallel migration changes are responsible for that, but I can find such records in my log days before the change.

System Setup

• List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount".
• OS version: Debian 10
• output of weed version: version 30GB 2.88 7270067289802307556d117be422a1e5a208f558 linux amd64

[divanikus]

@chrislusf What do you think, could this be caused by double incrementing the volume count?

https://github.com/chrislusf/seaweedfs/blob/master/weed/shell/command_volume_tier_move.go#L205
https://github.com/chrislusf/seaweedfs/blob/master/weed/shell/command_volume_tier_move.go#L238

[chrislusf]

@chrislusf What do you think, could this be caused by double incrementing the volume count?

https://github.com/chrislusf/seaweedfs/blob/master/weed/shell/command_volume_tier_move.go#L205 https://github.com/chrislusf/seaweedfs/blob/master/weed/shell/command_volume_tier_move.go#L238

This may be a separate issue. I will address this.