Even though Garbage collection running automatically disk space is not reduced.

[rgaduput]

We have SeaweedFS running in Kubernetes as separate components (master, filer, volume).
There is a bucket in which files are uploaded and removed periodically.
I see that in master logs its running the vacuum process automatically but still the disk space is not released.
Currently in the bucket we have about 17 files with each of being about size 180M. But on the disk level 60G is used.
Any idea why the vacuum does not perform garbage collection and release disk ?

Bucket items

ls buckets/sm-backup
sm-backup-2024-07-27-14-00-sm.tgz
sm-backup-2024-07-27-16-00-sm.tgz
sm-backup-2024-07-27-20-00-sm.tgz
sm-backup-2024-07-28-00-00-sm.tgz
sm-backup-2024-07-28-04-00-sm.tgz
sm-backup-2024-07-28-06-00-sm.tgz
sm-backup-2024-07-28-08-00-sm.tgz
sm-backup-2024-07-28-12-00-sm.tgz
sm-backup-2024-07-28-14-00-sm.tgz
sm-backup-2024-07-28-18-00-sm.tgz
sm-backup-2024-07-28-22-00-sm.tgz
sm-backup-2024-07-29-00-00-sm.tgz
sm-backup-2024-07-29-02-00-sm.tgz
sm-backup-2024-07-29-04-00-sm.tgz
sm-backup-2024-07-29-08-00-sm.tgz
sm-backup-2024-07-29-10-00-sm.tgz
sm-backup-2024-07-29-12-00-sm.tgz

Disk size on Volume server

Filesystem                Size      Used Available Use% Mounted on
/dev/nvme5n1             78.6G     56.2G     22.4G  72% /data
/data #

Items on Disk level

/data # du -sh * | grep -v K
2.4M    10.dat
2.1M    11.dat
2.6M    12.dat
2.9M    7.dat
3.0M    8.dat
2.5M    9.dat
1005.2M sm-backup_1.dat
1016.5M sm-backup_13.dat
1001.9M sm-backup_14.dat
1001.9M sm-backup_15.dat
1007.9M sm-backup_16.dat
1022.3M sm-backup_17.dat
1009.8M sm-backup_18.dat
1002.3M sm-backup_19.dat
1009.5M sm-backup_2.dat
1016.3M sm-backup_20.dat
1002.9M sm-backup_21.dat
1001.0M sm-backup_22.dat
1017.8M sm-backup_23.dat
1008.8M sm-backup_24.dat
1007.1M sm-backup_3.dat
1004.7M sm-backup_37.dat
1010.6M sm-backup_38.dat
1022.8M sm-backup_39.dat
1008.4M sm-backup_4.dat
1.0G    sm-backup_40.dat
1.0G    sm-backup_41.dat
1003.0M sm-backup_42.dat
1013.9M sm-backup_43.dat
1007.8M sm-backup_44.dat
1001.6M sm-backup_45.dat
1008.7M sm-backup_46.dat
1003.5M sm-backup_47.dat
1000.4M sm-backup_48.dat
1001.7M sm-backup_49.dat
1013.5M sm-backup_5.dat
1.0G    sm-backup_50.dat
1.0G    sm-backup_51.dat
1006.4M sm-backup_52.dat
1004.3M sm-backup_53.dat
1023.1M sm-backup_54.dat
1006.9M sm-backup_55.dat
1015.8M sm-backup_56.dat
1.0G    sm-backup_57.dat
1022.7M sm-backup_58.dat
1010.4M sm-backup_59.dat
1011.7M sm-backup_6.dat
1007.2M sm-backup_60.dat
1.0G    sm-backup_61.dat
1021.7M sm-backup_62.dat
1009.2M sm-backup_63.dat
1.0G    sm-backup_64.dat
1.0G    sm-backup_65.dat
854.5M  sm-backup_66.dat
1.0G    sm-backup_67.dat
1.0G    sm-backup_68.dat
1014.2M sm-backup_69.dat
1.0G    sm-backup_70.dat
922.6M  sm-backup_71.dat
725.6M  sm-backup_72.dat
1.0G    sm-backup_73.dat
326.0M  sm-backup_74.dat
379.0M  sm-backup_75.dat
358.0M  sm-backup_76.dat
387.0M  sm-backup_77.dat
381.4M  sm-backup_78.dat
369.4M  sm-backup_79.dat

Master logs

I0729 14:22:02.719624 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:23
I0729 14:22:02.721033 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:70
I0729 14:22:02.722538 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:78
I0729 14:22:02.724026 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:1
I0729 14:22:02.725837 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:56
I0729 14:22:02.727486 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:69
I0729 14:22:02.729011 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:62
I0729 14:22:02.730616 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:57
I0729 14:22:02.732266 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:54
I0729 14:22:02.733955 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:77
I0729 14:22:02.735481 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:67
I0729 14:22:02.737032 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:68
I0729 14:22:02.738963 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:38
I0729 14:22:02.740582 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:37
I0729 14:22:02.742170 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:72
I0729 14:22:02.743883 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:53
I0729 14:22:02.746836 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:60
I0729 14:22:02.748574 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:17
I0729 14:22:02.750143 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:63
I0729 14:22:02.751665 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:2
I0729 14:22:02.753274 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:18
I0729 14:22:02.754901 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:55
I0729 14:22:02.756842 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:40
I0729 14:22:02.758562 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:75
I0729 14:22:02.760184 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:16
I0729 14:22:02.762147 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:76
I0729 14:22:02.770184 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:4
I0729 14:22:02.772443 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:74
I0729 14:22:02.775180 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:13
I0729 14:22:02.776932 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:59
I0729 14:22:02.778710 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:50
I0729 14:22:02.780343 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:79
I0729 14:22:02.782001 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:64
I0729 14:22:02.783586 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:20
I0729 14:22:02.785225 topology_vacuum.go:277 check vacuum on collection:sm-backup volume:19
I0729 14:22:02.786767 topology_vacuum.go:277 check vacuum on collection: volume:9
I0729 14:22:02.788219 topology_vacuum.go:277 check vacuum on collection: volume:10
I0729 14:22:02.789828 topology_vacuum.go:277 check vacuum on collection: volume:11
I0729 14:22:02.791387 topology_vacuum.go:277 check vacuum on collection: volume:12
I0729 14:22:02.792787 topology_vacuum.go:277 check vacuum on collection: volume:7
I0729 14:22:02.794399 topology_vacuum.go:277 check vacuum on collection: volume:8

volume.list output

> volume.list
Topology volumeSizeLimit:1000 MB hdd(volume:134/158 active:-24 free:24 remote:0)
  DataCenter DefaultDataCenter hdd(volume:134/158 active:-24 free:24 remote:0)
    Rack DefaultRack hdd(volume:134/158 active:-24 free:24 remote:0)
      DataNode seaweedfs-volume-0.seaweedfs-volume.management:8080 hdd(volume:67/79 active:-12 free:12 remote:0)
        Disk hdd(volume:67/79 active:-12 free:12 remote:0)
          volume id:1  size:1053974432  collection:"sm-backup"  file_count:405  delete_count:82  deleted_byte_count:186648304  replica_placement:1  version:3  compact_revision:157  modified_at_second:1721978467
          volume id:2  size:1058491904  collection:"sm-backup"  file_count:407  delete_count:42  deleted_byte_count:122684310  replica_placement:1  version:3  compact_revision:164  modified_at_second:1722148849
          volume id:3  size:1055992168  collection:"sm-backup"  file_count:394  delete_count:116  deleted_byte_count:302631089  replica_placement:1  version:3  compact_revision:169  modified_at_second:1721735829
          volume id:4  size:1057371952  collection:"sm-backup"  file_count:407  delete_count:65  deleted_byte_count:178259303  replica_placement:1  version:3  compact_revision:161  modified_at_second:1722148849
          volume id:5  size:1062717008  collection:"sm-backup"  file_count:409  delete_count:66  deleted_byte_count:144704910  replica_placement:1  version:3  compact_revision:174  modified_at_second:1722148850
          volume id:6  size:1060848336  collection:"sm-backup"  file_count:400  delete_count:112  deleted_byte_count:290457922  replica_placement:1  version:3  compact_revision:173  modified_at_second:1721735829
          volume id:7  size:2994272  file_count:567  replica_placement:1  version:3  modified_at_second:1722254564
          volume id:8  size:3095712  file_count:572  replica_placement:1  version:3  modified_at_second:1722240179
          volume id:9  size:2586232  file_count:573  replica_placement:1  version:3  modified_at_second:1722232979
          volume id:10  size:2512952  file_count:562  replica_placement:1  version:3  modified_at_second:1722254519
          volume id:11  size:2163728  file_count:560  replica_placement:1  version:3  modified_at_second:1722261719
          volume id:12  size:2699808  file_count:533  replica_placement:1  version:3  modified_at_second:1722261764
          volume id:13  size:1065905720  collection:"sm-backup"  file_count:397  delete_count:65  deleted_byte_count:173617449  replica_placement:1  version:3  compact_revision:2  modified_at_second:1721978467
          volume id:14  size:1050581872  collection:"sm-backup"  file_count:399  delete_count:77  deleted_byte_count:200871177  replica_placement:1  version:3  compact_revision:2  modified_at_second:1721978467
          volume id:15  size:1050558960  collection:"sm-backup"  file_count:405  delete_count:105  deleted_byte_count:257172572  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:16  size:1056862584  collection:"sm-backup"  file_count:414  delete_count:109  deleted_byte_count:287312167  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:17  size:1071954248  collection:"sm-backup"  file_count:406  delete_count:99  deleted_byte_count:288879706  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:18  size:1058893888  collection:"sm-backup"  file_count:388  delete_count:90  deleted_byte_count:279971700  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:19  size:1050964560  collection:"sm-backup"  file_count:402  delete_count:107  deleted_byte_count:288360683  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:20  size:1065694200  collection:"sm-backup"  file_count:388  delete_count:91  deleted_byte_count:246417271  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:21  size:1051559552  collection:"sm-backup"  file_count:395  delete_count:88  deleted_byte_count:236980060  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059090
          volume id:22  size:1049613232  collection:"sm-backup"  file_count:414  delete_count:102  deleted_byte_count:257951856  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:23  size:1067269656  collection:"sm-backup"  file_count:409  delete_count:89  deleted_byte_count:212862833  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:24  size:1057754248  collection:"sm-backup"  file_count:411  delete_count:96  deleted_byte_count:251660310  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:37  size:1053547824  collection:"sm-backup"  file_count:419  delete_count:113  deleted_byte_count:265863971  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:38  size:1059708176  collection:"sm-backup"  file_count:407  delete_count:116  deleted_byte_count:297798020  replica_placement:1  version:3  compact_revision:1  modified_at_second:1721978467
          volume id:39  size:1072519536  collection:"sm-backup"  file_count:403  delete_count:107  deleted_byte_count:272632007  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:40  size:1084301832  collection:"sm-backup"  file_count:412  delete_count:41  deleted_byte_count:115344221  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722234863
          volume id:41  size:1074937408  collection:"sm-backup"  file_count:401  delete_count:100  deleted_byte_count:278709757  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:42  size:1051668336  collection:"sm-backup"  file_count:417  delete_count:120  deleted_byte_count:295701006  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:43  size:1063114776  collection:"sm-backup"  file_count:380  delete_count:90  deleted_byte_count:249641079  replica_placement:1  version:3  compact_revision:1  modified_at_second:1721978467
          volume id:44  size:1056713184  collection:"sm-backup"  file_count:413  delete_count:101  deleted_byte_count:272655589  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:45  size:1050261800  collection:"sm-backup"  file_count:412  delete_count:108  deleted_byte_count:279972078  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059090
          volume id:46  size:1057701216  collection:"sm-backup"  file_count:420  delete_count:100  deleted_byte_count:227543164  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:47  size:1052197128  collection:"sm-backup"  file_count:416  delete_count:92  deleted_byte_count:219154334  replica_placement:1  version:3  compact_revision:1  modified_at_second:1721978467
          volume id:48  size:1048977376  collection:"sm-backup"  file_count:406  delete_count:102  deleted_byte_count:270534768  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:49  size:1050370024  collection:"sm-backup"  file_count:391  delete_count:96  deleted_byte_count:279971826  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:50  size:1096733632  collection:"sm-backup"  file_count:433  delete_count:112  deleted_byte_count:277875146  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148890
          volume id:51  size:1081475752  collection:"sm-backup"  file_count:421  delete_count:99  deleted_byte_count:254806065  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148890
          volume id:52  size:1055285680  collection:"sm-backup"  file_count:383  delete_count:91  deleted_byte_count:255854509  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:53  size:1053056696  collection:"sm-backup"  file_count:403  delete_count:103  deleted_byte_count:271583365  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148890
          volume id:54  size:1072829304  collection:"sm-backup"  file_count:402  delete_count:102  deleted_byte_count:279472379  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148890
          volume id:55  size:1055830024  collection:"sm-backup"  file_count:412  delete_count:89  deleted_byte_count:225445709  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:56  size:1065162752  collection:"sm-backup"  file_count:391  delete_count:111  deleted_byte_count:305137947  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:57  size:1093260912  collection:"sm-backup"  file_count:410  delete_count:16  deleted_byte_count:40001231  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722254465
          volume id:58  size:1072334304  collection:"sm-backup"  file_count:410  delete_count:114  deleted_byte_count:311429502  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:59  size:1059481808  collection:"sm-backup"  file_count:419  delete_count:124  deleted_byte_count:306186796  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:60  size:1056132752  collection:"sm-backup"  file_count:401  delete_count:111  deleted_byte_count:305137965  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:61  size:1073860048  collection:"sm-backup"  file_count:414  delete_count:14  deleted_byte_count:36700472  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722254465
          volume id:62  size:1071356280  collection:"sm-backup"  file_count:408  delete_count:96  deleted_byte_count:254805984  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722234861
          volume id:63  size:1058247088  collection:"sm-backup"  file_count:427  delete_count:120  deleted_byte_count:273680856  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722234863
          volume id:64  size:1081294696  collection:"sm-backup"  file_count:396  delete_count:84  deleted_byte_count:223348470  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722234861
          volume id:65  size:1080232936  collection:"sm-backup"  file_count:398  delete_count:47  deleted_byte_count:118490093  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722254465
          volume id:66  size:896041008  collection:"sm-backup"  file_count:344  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722261718
          volume id:67  size:1084204032  collection:"sm-backup"  file_count:425  delete_count:118  deleted_byte_count:303040996  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722234863
          volume id:68  size:1096550456  collection:"sm-backup"  file_count:423  delete_count:108  deleted_byte_count:270534894  replica_placement:1  version:3  modified_at_second:1722254465
          volume id:69  size:1063436856  collection:"sm-backup"  file_count:402  delete_count:105  deleted_byte_count:276826341  replica_placement:1  version:3  modified_at_second:1722254465
          volume id:70  size:1109323440  collection:"sm-backup"  file_count:427  delete_count:107  deleted_byte_count:282069191  replica_placement:1  version:3  modified_at_second:1722254465
          volume id:71  size:967383816  collection:"sm-backup"  file_count:363  delete_count:4  deleted_byte_count:13631572  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722261718
          volume id:72  size:760792872  collection:"sm-backup"  file_count:293  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722261718
          volume id:73  size:1080060248  collection:"sm-backup"  file_count:412  delete_count:119  deleted_byte_count:310381031  replica_placement:1  version:3  modified_at_second:1722254465
          volume id:74  size:341825864  collection:"sm-backup"  file_count:142  replica_placement:1  version:3  modified_at_second:1722261717
          volume id:75  size:397418664  collection:"sm-backup"  file_count:148  replica_placement:1  version:3  modified_at_second:1722261718
          volume id:76  size:375398704  collection:"sm-backup"  file_count:151  replica_placement:1  version:3  modified_at_second:1722261718
          volume id:77  size:405807032  collection:"sm-backup"  file_count:144  replica_placement:1  version:3  modified_at_second:1722261718
          volume id:78  size:399891928  collection:"sm-backup"  file_count:166  replica_placement:1  version:3  modified_at_second:1722261718
          volume id:79  size:387291752  collection:"sm-backup"  file_count:147  replica_placement:1  version:3  modified_at_second:1722261718
        Disk hdd total size:60331081176 file_count:26429 deleted_file:4881 deleted_bytes:12729405956
      DataNode seaweedfs-volume-0.seaweedfs-volume.management:8080 total size:60331081176 file_count:26429 deleted_file:4881 deleted_bytes:12729405956
      DataNode seaweedfs-volume-1.seaweedfs-volume.management:8080 hdd(volume:67/79 active:-12 free:12 remote:0)
        Disk hdd(volume:67/79 active:-12 free:12 remote:0)
          volume id:1  size:1053974432  collection:"sm-backup"  file_count:405  delete_count:82  deleted_byte_count:186648304  replica_placement:1  version:3  compact_revision:157  modified_at_second:1721978467
          volume id:2  size:1058491904  collection:"sm-backup"  file_count:407  delete_count:42  deleted_byte_count:122684310  replica_placement:1  version:3  compact_revision:164  modified_at_second:1722148849
          volume id:3  size:1055992168  collection:"sm-backup"  file_count:394  delete_count:116  deleted_byte_count:302631089  replica_placement:1  version:3  compact_revision:169  modified_at_second:1721735829
          volume id:4  size:1057371952  collection:"sm-backup"  file_count:407  delete_count:65  deleted_byte_count:178259303  replica_placement:1  version:3  compact_revision:161  modified_at_second:1722148849
          volume id:5  size:1062717008  collection:"sm-backup"  file_count:409  delete_count:66  deleted_byte_count:144704910  replica_placement:1  version:3  compact_revision:174  modified_at_second:1722148850
          volume id:6  size:1060848336  collection:"sm-backup"  file_count:400  delete_count:112  deleted_byte_count:290457922  replica_placement:1  version:3  compact_revision:173  modified_at_second:1721735829
          volume id:7  size:2994272  file_count:567  replica_placement:1  version:3  modified_at_second:1722254564
          volume id:8  size:3095712  file_count:572  replica_placement:1  version:3  modified_at_second:1722240179
          volume id:9  size:2586232  file_count:573  replica_placement:1  version:3  modified_at_second:1722232979
          volume id:10  size:2512952  file_count:562  replica_placement:1  version:3  modified_at_second:1722254519
          volume id:11  size:2163728  file_count:560  replica_placement:1  version:3  modified_at_second:1722261719
          volume id:12  size:2699808  file_count:533  replica_placement:1  version:3  modified_at_second:1722261764
          volume id:13  size:1065905720  collection:"sm-backup"  file_count:397  delete_count:65  deleted_byte_count:173617449  replica_placement:1  version:3  compact_revision:2  modified_at_second:1721978467
          volume id:14  size:1050581872  collection:"sm-backup"  file_count:399  delete_count:77  deleted_byte_count:200871177  replica_placement:1  version:3  compact_revision:2  modified_at_second:1721978467
          volume id:15  size:1050558960  collection:"sm-backup"  file_count:405  delete_count:105  deleted_byte_count:257172572  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:16  size:1056862584  collection:"sm-backup"  file_count:414  delete_count:109  deleted_byte_count:287312167  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:17  size:1071954248  collection:"sm-backup"  file_count:406  delete_count:99  deleted_byte_count:288879706  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:18  size:1058893888  collection:"sm-backup"  file_count:388  delete_count:90  deleted_byte_count:279971700  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:19  size:1050964560  collection:"sm-backup"  file_count:402  delete_count:107  deleted_byte_count:288360683  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:20  size:1065694200  collection:"sm-backup"  file_count:388  delete_count:91  deleted_byte_count:246417271  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:21  size:1051559552  collection:"sm-backup"  file_count:395  delete_count:88  deleted_byte_count:236980060  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059090
          volume id:22  size:1049613232  collection:"sm-backup"  file_count:414  delete_count:102  deleted_byte_count:257951856  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:23  size:1067269656  collection:"sm-backup"  file_count:409  delete_count:89  deleted_byte_count:212862833  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:24  size:1057754248  collection:"sm-backup"  file_count:411  delete_count:96  deleted_byte_count:251660310  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:37  size:1053547824  collection:"sm-backup"  file_count:419  delete_count:113  deleted_byte_count:265863971  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:38  size:1059708176  collection:"sm-backup"  file_count:407  delete_count:116  deleted_byte_count:297798020  replica_placement:1  version:3  compact_revision:1  modified_at_second:1721978467
          volume id:39  size:1072519536  collection:"sm-backup"  file_count:403  delete_count:107  deleted_byte_count:272632007  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:40  size:1084301832  collection:"sm-backup"  file_count:412  delete_count:41  deleted_byte_count:115344221  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722234863
          volume id:41  size:1074937408  collection:"sm-backup"  file_count:401  delete_count:100  deleted_byte_count:278709757  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:42  size:1051668336  collection:"sm-backup"  file_count:417  delete_count:120  deleted_byte_count:295701006  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:43  size:1063114776  collection:"sm-backup"  file_count:380  delete_count:90  deleted_byte_count:249641079  replica_placement:1  version:3  compact_revision:1  modified_at_second:1721978467
          volume id:44  size:1056713184  collection:"sm-backup"  file_count:413  delete_count:101  deleted_byte_count:272655589  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:45  size:1050261800  collection:"sm-backup"  file_count:412  delete_count:108  deleted_byte_count:279972078  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059090
          volume id:46  size:1057701216  collection:"sm-backup"  file_count:420  delete_count:100  deleted_byte_count:227543164  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059088
          volume id:47  size:1052197128  collection:"sm-backup"  file_count:416  delete_count:92  deleted_byte_count:219154334  replica_placement:1  version:3  compact_revision:1  modified_at_second:1721978467
          volume id:48  size:1048977376  collection:"sm-backup"  file_count:406  delete_count:102  deleted_byte_count:270534768  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:49  size:1050370024  collection:"sm-backup"  file_count:391  delete_count:96  deleted_byte_count:279971826  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722059089
          volume id:50  size:1096733632  collection:"sm-backup"  file_count:433  delete_count:112  deleted_byte_count:277875146  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148890
          volume id:51  size:1081475752  collection:"sm-backup"  file_count:421  delete_count:99  deleted_byte_count:254806065  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148890
          volume id:52  size:1055285680  collection:"sm-backup"  file_count:383  delete_count:91  deleted_byte_count:255854509  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:53  size:1053056696  collection:"sm-backup"  file_count:403  delete_count:103  deleted_byte_count:271583365  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148890
          volume id:54  size:1072829304  collection:"sm-backup"  file_count:402  delete_count:102  deleted_byte_count:279472379  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148890
          volume id:55  size:1055830024  collection:"sm-backup"  file_count:412  delete_count:89  deleted_byte_count:225445709  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:56  size:1065162752  collection:"sm-backup"  file_count:391  delete_count:111  deleted_byte_count:305137947  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:57  size:1093260912  collection:"sm-backup"  file_count:410  delete_count:16  deleted_byte_count:40001231  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722254465
          volume id:58  size:1072334304  collection:"sm-backup"  file_count:410  delete_count:114  deleted_byte_count:311429502  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:59  size:1059481808  collection:"sm-backup"  file_count:419  delete_count:124  deleted_byte_count:306186796  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:60  size:1056132752  collection:"sm-backup"  file_count:401  delete_count:111  deleted_byte_count:305137965  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722148850
          volume id:61  size:1073860048  collection:"sm-backup"  file_count:414  delete_count:14  deleted_byte_count:36700472  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722254465
          volume id:62  size:1071356280  collection:"sm-backup"  file_count:408  delete_count:96  deleted_byte_count:254805984  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722234861
          volume id:63  size:1058247088  collection:"sm-backup"  file_count:427  delete_count:120  deleted_byte_count:273680856  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722234863
          volume id:64  size:1081294696  collection:"sm-backup"  file_count:396  delete_count:84  deleted_byte_count:223348470  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722234861
          volume id:65  size:1080232936  collection:"sm-backup"  file_count:398  delete_count:47  deleted_byte_count:118490093  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722254465
          volume id:66  size:896041008  collection:"sm-backup"  file_count:344  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722261718
          volume id:67  size:1084204032  collection:"sm-backup"  file_count:425  delete_count:118  deleted_byte_count:303040996  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722234863
          volume id:68  size:1096550456  collection:"sm-backup"  file_count:423  delete_count:108  deleted_byte_count:270534894  replica_placement:1  version:3  modified_at_second:1722254465
          volume id:69  size:1063436856  collection:"sm-backup"  file_count:402  delete_count:105  deleted_byte_count:276826341  replica_placement:1  version:3  modified_at_second:1722254465
          volume id:70  size:1109323440  collection:"sm-backup"  file_count:427  delete_count:107  deleted_byte_count:282069191  replica_placement:1  version:3  modified_at_second:1722254465
          volume id:71  size:967383816  collection:"sm-backup"  file_count:363  delete_count:4  deleted_byte_count:13631572  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722261718
          volume id:72  size:760792872  collection:"sm-backup"  file_count:293  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722261718
          volume id:73  size:1080060248  collection:"sm-backup"  file_count:412  delete_count:119  deleted_byte_count:310381031  replica_placement:1  version:3  modified_at_second:1722254465
          volume id:74  size:341825864  collection:"sm-backup"  file_count:142  replica_placement:1  version:3  modified_at_second:1722261717
          volume id:75  size:397418664  collection:"sm-backup"  file_count:148  replica_placement:1  version:3  modified_at_second:1722261718
          volume id:76  size:375398704  collection:"sm-backup"  file_count:151  replica_placement:1  version:3  modified_at_second:1722261718
          volume id:77  size:405807032  collection:"sm-backup"  file_count:144  replica_placement:1  version:3  modified_at_second:1722261718
          volume id:78  size:399891928  collection:"sm-backup"  file_count:166  replica_placement:1  version:3  modified_at_second:1722261718
          volume id:79  size:387291752  collection:"sm-backup"  file_count:147  replica_placement:1  version:3  modified_at_second:1722261718
        Disk hdd total size:60331081176 file_count:26429 deleted_file:4881 deleted_bytes:12729405956
      DataNode seaweedfs-volume-1.seaweedfs-volume.management:8080 total size:60331081176 file_count:26429 deleted_file:4881 deleted_bytes:12729405956
    Rack DefaultRack total size:120662162352 file_count:52858 deleted_file:9762 deleted_bytes:25458811912
  DataCenter DefaultDataCenter total size:120662162352 file_count:52858 deleted_file:9762 deleted_bytes:25458811912
total size:120662162352 file_count:52858 deleted_file:9762 deleted_bytes:25458811912

Appreciate any suggestions. Thanks.

[chrislusf]

• What is the version?
• What's the master.toml content?
• what's the output of

ls -l buckets/sm-backup

[rgaduput]

@chrislusf
Version: 3.55 7eafa3420 linux amd64
master.toml does not exists but below is the command used in master

/usr/bin/weed  -logtostderr=true  -v=1  master  -port=9333  -mdir=/data  -ip.bind=0.0.0.0  -defaultReplication=001  -volumeSizeLimitMB=1000  -ip=${POD_NAME}.${SEAWEEDFS_FULLNAME}-master.management  -peers=${SEAWEEDFS_FULLNAME}-master-0.${SEAWEEDFS_FULLNAME}-master.management:9333,${SEAWEEDFS_FULLNAME}-master-1.${SEAWEEDFS_FULLNAME}-master.management:9333,${SEAWEEDFS_FULLNAME}-master-2.${SEAWEEDFS_FULLNAME}-master.management:9333

Master pod env vars:

SEAWEEDFS_FULLNAME : seaweedfs
WEED_CLUSTER_DEFAULT : sw
WEED_CLUSTER_SW_FILER : seaweedfs-filer-client.seaweedfs:8888
WEED_CLUSTER_SW_MASTER : seaweedfs-master.seaweedfs:9333
WEED_MASTER_VOLUME_GROWTH_COPY_1 : 7
WEED_MASTER_VOLUME_GROWTH_COPY_2 : 6
WEED_MASTER_VOLUME_GROWTH_COPY_3 : 3
WEED_MASTER_VOLUME_GROWTH_COPY_OTHER : 1

Output of ls -l buckets/sm-backup

> ls -l buckets/sm-backup
-rwxrwx---  69 root root 180071484 /buckets/sm-backup/sm-backup-2024-07-27-14-00-sm.tgz
-rwxrwx---  70 root root 183500800 /buckets/sm-backup/sm-backup-2024-07-27-16-00-sm.tgz
-rwxrwx---  70 root root 183500800 /buckets/sm-backup/sm-backup-2024-07-27-20-00-sm.tgz
-rwxrwx---  70 root root 183500800 /buckets/sm-backup/sm-backup-2024-07-28-00-00-sm.tgz
-rwxrwx---  70 root root 183500800 /buckets/sm-backup/sm-backup-2024-07-28-04-00-sm.tgz
-rwxrwx---  70 root root 183500800 /buckets/sm-backup/sm-backup-2024-07-28-06-00-sm.tgz
-rwxrwx---  70 root root 183500800 /buckets/sm-backup/sm-backup-2024-07-28-08-00-sm.tgz
-rwxrwx---  70 root root 183500800 /buckets/sm-backup/sm-backup-2024-07-28-12-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-28-14-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-28-18-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-28-22-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-29-00-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-29-02-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-29-04-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-29-08-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-29-10-00-sm.tgz
-rwxrwx---  72 root root 188743680 /buckets/sm-backup/sm-backup-2024-07-29-12-00-sm.tgz
-rwxrwx---  74 root root 193986560 /buckets/sm-backup/sm-backup-2024-07-29-16-00-sm.tgz
-rwxrwx---  74 root root 193986560 /buckets/sm-backup/sm-backup-2024-07-30-06-00-sm.tgz
total 19
>

Don't know if its relevant but files removal from bucket was done by using S3 API (s3cmd) and not weed shell

[chrislusf]

may need to compact the volumes with a lower threshold.

Can you show the volume.list output and also volume.fsck output?

[rgaduput]

volume.list

> volume.list
Topology volumeSizeLimit:1000 MB hdd(volume:134/158 active:126 free:24 remote:0)
  DataCenter DefaultDataCenter hdd(volume:134/158 active:126 free:24 remote:0)
    Rack DefaultRack hdd(volume:134/158 active:126 free:24 remote:0)
      DataNode seaweedfs-volume-0.seaweedfs-volume.secure-management:8080 hdd(volume:67/79 active:63 free:12 remote:0)
        Disk hdd(volume:67/79 active:63 free:12 remote:0)
          volume id:1  size:415526368  collection:"sm-backup"  file_count:160  replica_placement:1  version:3  compact_revision:158  modified_at_second:1722410890
          volume id:2  size:446165520  collection:"sm-backup"  file_count:182  replica_placement:1  version:3  compact_revision:165  modified_at_second:1722411302
          volume id:3  size:347242600  collection:"sm-backup"  file_count:123  replica_placement:1  version:3  compact_revision:170  modified_at_second:1722411473
          volume id:4  size:398434424  collection:"sm-backup"  file_count:159  replica_placement:1  version:3  compact_revision:162  modified_at_second:1722411168
          volume id:5  size:497447184  collection:"sm-backup"  file_count:182  replica_placement:1  version:3  compact_revision:175  modified_at_second:1722411263
          volume id:6  size:337374032  collection:"sm-backup"  file_count:130  replica_placement:1  version:3  compact_revision:174  modified_at_second:1722411072
          volume id:7  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:8  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:9  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:10  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:11  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:12  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:13  size:373264624  collection:"sm-backup"  file_count:143  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411177
          volume id:14  size:364735712  collection:"sm-backup"  file_count:141  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722410971
          volume id:15  size:330368288  collection:"sm-backup"  file_count:127  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411041
          volume id:16  size:341264376  collection:"sm-backup"  file_count:131  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410916
          volume id:17  size:312030600  collection:"sm-backup"  file_count:125  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411185
          volume id:18  size:368085936  collection:"sm-backup"  file_count:141  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411222
          volume id:19  size:376196880  collection:"sm-backup"  file_count:145  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411103
          volume id:20  size:450714024  collection:"sm-backup"  file_count:156  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411241
          volume id:21  size:467904416  collection:"sm-backup"  file_count:174  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411083
          volume id:22  size:367339176  collection:"sm-backup"  file_count:154  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410856
          volume id:23  size:428395984  collection:"sm-backup"  file_count:153  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411382
          volume id:24  size:373678296  collection:"sm-backup"  file_count:149  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411049
          volume id:37  size:375757408  collection:"sm-backup"  file_count:152  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410925
          volume id:38  size:376365368  collection:"sm-backup"  file_count:143  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411022
          volume id:39  size:427795512  collection:"sm-backup"  file_count:157  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410935
          volume id:40  size:457708936  collection:"sm-backup"  file_count:181  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411432
          volume id:41  size:439697160  collection:"sm-backup"  file_count:159  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411093
          volume id:42  size:407781056  collection:"sm-backup"  file_count:157  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411213
          volume id:43  size:420487272  collection:"sm-backup"  file_count:146  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411032
          volume id:44  size:376145920  collection:"sm-backup"  file_count:154  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411193
          volume id:45  size:422352968  collection:"sm-backup"  file_count:159  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411012
          volume id:46  size:442336904  collection:"sm-backup"  file_count:167  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410946
          volume id:47  size:360487272  collection:"sm-backup"  file_count:147  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411158
          volume id:48  size:344196280  collection:"sm-backup"  file_count:145  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411231
          volume id:49  size:440352352  collection:"sm-backup"  file_count:163  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410982
          volume id:50  size:407845440  collection:"sm-backup"  file_count:160  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410992
          volume id:51  size:403551424  collection:"sm-backup"  file_count:162  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411274
          volume id:52  size:404084728  collection:"sm-backup"  file_count:154  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411002
          volume id:53  size:389388600  collection:"sm-backup"  file_count:151  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411112
          volume id:54  size:363090704  collection:"sm-backup"  file_count:136  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411324
          volume id:55  size:399149560  collection:"sm-backup"  file_count:153  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411251
          volume id:56  size:493341072  collection:"sm-backup"  file_count:183  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410879
          volume id:57  size:719609848  collection:"sm-backup"  file_count:264  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411340
          volume id:58  size:512902536  collection:"sm-backup"  file_count:201  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410867
          volume id:59  size:532692256  collection:"sm-backup"  file_count:197  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411136
          volume id:60  size:495374184  collection:"sm-backup"  file_count:193  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411148
          volume id:61  size:709310048  collection:"sm-backup"  file_count:273  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411462
          volume id:62  size:663451496  collection:"sm-backup"  file_count:238  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722410906
          volume id:63  size:603884744  collection:"sm-backup"  file_count:239  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722410961
          volume id:64  size:658509680  collection:"sm-backup"  file_count:251  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411357
          volume id:65  size:539154944  collection:"sm-backup"  file_count:197  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411062
          volume id:66  size:393741808  collection:"sm-backup"  file_count:153  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411203
          volume id:67  size:662810360  collection:"sm-backup"  file_count:258  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411289
          volume id:68  size:1011167328  collection:"sm-backup"  file_count:397  delete_count:98  deleted_byte_count:262515411  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722410848
          volume id:69  size:1062265800  collection:"sm-backup"  file_count:400  delete_count:103  deleted_byte_count:285857550  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722410848
          volume id:70  size:1058991216  collection:"sm-backup"  file_count:415  delete_count:110  deleted_byte_count:283661163  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722410848
          volume id:71  size:929637976  collection:"sm-backup"  file_count:357  delete_count:105  deleted_byte_count:273339067  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410848
          volume id:72  size:526840600  collection:"sm-backup"  file_count:196  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411420
          volume id:73  size:439362584  collection:"sm-backup"  file_count:164  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411123
          volume id:74  size:505840576  collection:"sm-backup"  file_count:197  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411394
          volume id:75  size:516419680  collection:"sm-backup"  file_count:193  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411314
          volume id:76  size:504376472  collection:"sm-backup"  file_count:202  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411407
          volume id:77  size:557864280  collection:"sm-backup"  file_count:198  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411445
          volume id:78  size:491418096  collection:"sm-backup"  file_count:200  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411371
          volume id:79  size:787394400  collection:"sm-backup"  file_count:288  delete_count:85  deleted_byte_count:236127981  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722410848
        Disk hdd total size:30031105336 file_count:11475 deleted_file:501 deleted_bytes:1341501172
      DataNode seaweedfs-volume-0.seaweedfs-volume.secure-management:8080 total size:30031105336 file_count:11475 deleted_file:501 deleted_bytes:1341501172
      DataNode seaweedfs-volume-1.seaweedfs-volume.secure-management:8080 hdd(volume:67/79 active:63 free:12 remote:0)
        Disk hdd(volume:67/79 active:63 free:12 remote:0)
          volume id:1  size:415526368  collection:"sm-backup"  file_count:160  replica_placement:1  version:3  compact_revision:158  modified_at_second:1722410890
          volume id:2  size:446165520  collection:"sm-backup"  file_count:182  replica_placement:1  version:3  compact_revision:165  modified_at_second:1722411302
          volume id:3  size:347242600  collection:"sm-backup"  file_count:123  replica_placement:1  version:3  compact_revision:170  modified_at_second:1722411473
          volume id:4  size:398434424  collection:"sm-backup"  file_count:159  replica_placement:1  version:3  compact_revision:162  modified_at_second:1722411168
          volume id:5  size:497447184  collection:"sm-backup"  file_count:182  replica_placement:1  version:3  compact_revision:175  modified_at_second:1722411263
          volume id:6  size:337374032  collection:"sm-backup"  file_count:130  replica_placement:1  version:3  compact_revision:174  modified_at_second:1722411071
          volume id:7  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:8  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:9  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:10  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:11  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:12  size:8  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722411475
          volume id:13  size:373264624  collection:"sm-backup"  file_count:143  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411177
          volume id:14  size:364735712  collection:"sm-backup"  file_count:141  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722410971
          volume id:15  size:330368288  collection:"sm-backup"  file_count:127  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411041
          volume id:16  size:341264376  collection:"sm-backup"  file_count:131  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410916
          volume id:17  size:312030600  collection:"sm-backup"  file_count:125  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411185
          volume id:18  size:368085936  collection:"sm-backup"  file_count:141  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411222
          volume id:19  size:376196880  collection:"sm-backup"  file_count:145  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411103
          volume id:20  size:450714024  collection:"sm-backup"  file_count:156  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411241
          volume id:21  size:467904416  collection:"sm-backup"  file_count:174  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411082
          volume id:22  size:367339176  collection:"sm-backup"  file_count:154  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410856
          volume id:23  size:428395984  collection:"sm-backup"  file_count:153  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411382
          volume id:24  size:373678296  collection:"sm-backup"  file_count:149  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411050
          volume id:37  size:375757408  collection:"sm-backup"  file_count:152  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410925
          volume id:38  size:376365368  collection:"sm-backup"  file_count:143  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411022
          volume id:39  size:427795512  collection:"sm-backup"  file_count:157  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410935
          volume id:40  size:457708936  collection:"sm-backup"  file_count:181  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411432
          volume id:41  size:439697160  collection:"sm-backup"  file_count:159  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411093
          volume id:42  size:407781056  collection:"sm-backup"  file_count:157  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411213
          volume id:43  size:420487272  collection:"sm-backup"  file_count:146  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411032
          volume id:44  size:376145920  collection:"sm-backup"  file_count:154  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411194
          volume id:45  size:422352968  collection:"sm-backup"  file_count:159  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411012
          volume id:46  size:442336904  collection:"sm-backup"  file_count:167  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410946
          volume id:47  size:360487272  collection:"sm-backup"  file_count:147  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411158
          volume id:48  size:344196280  collection:"sm-backup"  file_count:145  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411231
          volume id:49  size:440352352  collection:"sm-backup"  file_count:163  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410982
          volume id:50  size:407845440  collection:"sm-backup"  file_count:160  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410992
          volume id:51  size:403551424  collection:"sm-backup"  file_count:162  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411274
          volume id:52  size:404084728  collection:"sm-backup"  file_count:154  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411002
          volume id:53  size:389388600  collection:"sm-backup"  file_count:151  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411112
          volume id:54  size:363090704  collection:"sm-backup"  file_count:136  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411324
          volume id:55  size:399149560  collection:"sm-backup"  file_count:153  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411252
          volume id:56  size:493341072  collection:"sm-backup"  file_count:183  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410879
          volume id:57  size:719609848  collection:"sm-backup"  file_count:264  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411340
          volume id:58  size:512902536  collection:"sm-backup"  file_count:201  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410867
          volume id:59  size:532692256  collection:"sm-backup"  file_count:197  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411136
          volume id:60  size:495374184  collection:"sm-backup"  file_count:193  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411148
          volume id:61  size:709310048  collection:"sm-backup"  file_count:273  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411462
          volume id:62  size:663451496  collection:"sm-backup"  file_count:238  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722410904
          volume id:63  size:603884744  collection:"sm-backup"  file_count:239  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722410959
          volume id:64  size:658509680  collection:"sm-backup"  file_count:251  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411357
          volume id:65  size:539154944  collection:"sm-backup"  file_count:197  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411062
          volume id:66  size:393741808  collection:"sm-backup"  file_count:153  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411203
          volume id:67  size:662810360  collection:"sm-backup"  file_count:258  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411289
          volume id:68  size:1011167328  collection:"sm-backup"  file_count:397  delete_count:98  deleted_byte_count:262515411  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722410848
          volume id:69  size:1062265800  collection:"sm-backup"  file_count:400  delete_count:103  deleted_byte_count:285857550  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722410848
          volume id:70  size:1058991216  collection:"sm-backup"  file_count:415  delete_count:110  deleted_byte_count:283661163  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722410848
          volume id:71  size:929637976  collection:"sm-backup"  file_count:357  delete_count:105  deleted_byte_count:273339067  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722410848
          volume id:72  size:526840600  collection:"sm-backup"  file_count:196  replica_placement:1  version:3  compact_revision:3  modified_at_second:1722411420
          volume id:73  size:439362584  collection:"sm-backup"  file_count:164  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411123
          volume id:74  size:505840576  collection:"sm-backup"  file_count:197  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411394
          volume id:75  size:516419680  collection:"sm-backup"  file_count:193  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411314
          volume id:76  size:504376472  collection:"sm-backup"  file_count:202  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411407
          volume id:77  size:557864280  collection:"sm-backup"  file_count:198  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411445
          volume id:78  size:491418096  collection:"sm-backup"  file_count:200  replica_placement:1  version:3  compact_revision:2  modified_at_second:1722411371
          volume id:79  size:787394400  collection:"sm-backup"  file_count:288  delete_count:85  deleted_byte_count:236127981  replica_placement:1  version:3  compact_revision:1  modified_at_second:1722410848
        Disk hdd total size:30031105336 file_count:11475 deleted_file:501 deleted_bytes:1341501172
      DataNode seaweedfs-volume-1.seaweedfs-volume.secure-management:8080 total size:30031105336 file_count:11475 deleted_file:501 deleted_bytes:1341501172
    Rack DefaultRack total size:60062210672 file_count:22950 deleted_file:1002 deleted_bytes:2683002344
  DataCenter DefaultDataCenter total size:60062210672 file_count:22950 deleted_file:1002 deleted_bytes:2683002344
total size:60062210672 file_count:22950 deleted_file:1002 deleted_bytes:2683002344

volume.fsck

> volume.fsck
total 245 directories, 4843 files
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:62       entries:238     orphan:15       6.30%   53477691B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:67       entries:258     orphan:9        3.49%   25166031B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:75       entries:193     orphan:7        3.63%   19923091B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:68       entries:299     orphan:11       3.68%   27263207B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:64       entries:251     orphan:13       5.18%   26214673B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:76       entries:202     orphan:19       9.41%   41943457B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:63       entries:239     orphan:8        3.35%   20971688B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:77       entries:198     orphan:18       9.09%   56623482B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:79       entries:203     orphan:7        3.45%   16777363B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:78       entries:200     orphan:11       5.50%   26818767B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:69       entries:297     orphan:15       5.05%   37749051B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:74       entries:197     orphan:6        3.05%   15728766B
dataNode:seaweedfs-volume-0.seaweedfs-volume.secure-management:8080     volume:70       entries:305     orphan:10       3.28%   23068882B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:79       entries:203     orphan:7        3.45%   16777363B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:69       entries:297     orphan:15       5.05%   37749051B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:75       entries:193     orphan:7        3.63%   19923091B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:74       entries:197     orphan:6        3.05%   15728766B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:78       entries:200     orphan:11       5.50%   26818767B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:68       entries:299     orphan:11       3.68%   27263207B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:67       entries:258     orphan:9        3.49%   25166031B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:64       entries:251     orphan:13       5.18%   26214673B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:77       entries:198     orphan:18       9.09%   56623482B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:62       entries:238     orphan:15       6.30%   53477691B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:76       entries:202     orphan:19       9.41%   41943457B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:70       entries:305     orphan:10       3.28%   23068882B
dataNode:seaweedfs-volume-1.seaweedfs-volume.secure-management:8080     volume:63       entries:239     orphan:8        3.35%   20971688B

Total           entries:21948   orphan:298      1.36%   783452298B
This could be normal if multiple filers or no filers are used.

[rgaduput]

I have ran the vaccum with 0.2 threshold and after that ran volume.fsck -reallyDeleteFromVolume.
After that the disk size looks much better.

/data # df -h .
Filesystem                Size      Used Available Use% Mounted on
/dev/nvme5n1             78.6G      3.3G     75.2G   4% /data
/data #

[rgaduput]

@chrislusf Hi,
Because we were facing some strange issues we have decided to clean the whole bucket and recreate it (waited few hours for the files upload to happen and tried below).
But seems like that did not help.

When done ls -l buckets/sm-backup from seadweedfs-master-0 (follower) pod

/data # weed shell -master localhost:9333
I0802 06:12:24.522458 masterclient.go:209 master localhost:9333 redirected to leader seaweedfs-master-2.seaweedfs-master.secure-management:9333
.master: localhost:9333 filers: [10.201.128.122:8888 10.201.136.215:8888]
> ls -l buckets/sm-backup
-rwxrwx---  79 root root 205461715 /buckets/sm-backup/sm-backup-2024-08-01-18-00-sm.tgz
-rwxrwx---  80 root root 209715200 /buckets/sm-backup/sm-backup-2024-08-01-22-00-sm.tgz
-rwxrwx---  80 root root 209715200 /buckets/sm-backup/sm-backup-2024-08-02-00-00-sm.tgz
total 3

When performed on seaweedfs-master-2 (leader) pod

/data # weed shell -master localhost:9333
I0802 06:31:36.091089 masterclient.go:209 master localhost:9333 redirected to leader seaweedfs-master-2.seaweedfs-master.secure-management:9333
.master: localhost:9333 filers: [10.201.128.122:8888 10.201.136.215:8888]
> ls -l buckets/sm-backup
-rwxrwx--- 157 root root 412968735 /buckets/sm-backup/sm-backup-2024-08-01-14-00-sm.tgz
-rwxrwx---  79 root root 205461715 /buckets/sm-backup/sm-backup-2024-08-01-18-00-sm.tgz
-rwxrwx--- 159 root root 416485036 /buckets/sm-backup/sm-backup-2024-08-01-20-00-sm.tgz
-rwxrwx---  80 root root 209715200 /buckets/sm-backup/sm-backup-2024-08-01-22-00-sm.tgz
-rwxrwx---  80 root root 209715200 /buckets/sm-backup/sm-backup-2024-08-02-00-00-sm.tgz
-rwxrwx--- 160 root root 418844520 /buckets/sm-backup/sm-backup-2024-08-02-02-00-sm.tgz
-rwxrwx--- 160 root root 419262862 /buckets/sm-backup/sm-backup-2024-08-02-04-00-sm.tgz
-rwxrwx--- 161 root root 419751861 /buckets/sm-backup/sm-backup-2024-08-02-06-00-sm.tgz
total 9

When same was done using s3cmd from outside the kubernetes cluster by doing the port forward of seaweedfs-filer service (port 8333)

s3cmd ls s3://sm-backup -H
2024-08-01 14:02   393M  s3://sm-backup/sm-backup-2024-08-01-14-00-sm.tgz
2024-08-01 16:02   394M  s3://sm-backup/sm-backup-2024-08-01-16-00-sm.tgz
2024-08-01 18:02   195M  s3://sm-backup/sm-backup-2024-08-01-18-00-sm.tgz
2024-08-01 20:02   397M  s3://sm-backup/sm-backup-2024-08-01-20-00-sm.tgz
2024-08-01 22:01   200M  s3://sm-backup/sm-backup-2024-08-01-22-00-sm.tgz
2024-08-02 00:02   200M  s3://sm-backup/sm-backup-2024-08-02-00-00-sm.tgz
2024-08-02 02:02   399M  s3://sm-backup/sm-backup-2024-08-02-02-00-sm.tgz
2024-08-02 04:02   399M  s3://sm-backup/sm-backup-2024-08-02-04-00-sm.tgz
2024-08-02 06:02   400M  s3://sm-backup/sm-backup-2024-08-02-06-00-sm.tgz

Other strange issue we have observed that it randomly showed us a bucket called s2-backup when performed ls buckets But this bucket was completely deleted 15 days ago (all files + bucket was removed). It disappeared again after few minutes.

Other really concerning issue is that even though all files which are uploaded are like approx. 400M size as you can see some files show only half size of it inside bucket. When downloaded those for inspection observed that they are corrupted (file type is data and not tar/gzip).

any idea or suggestion to understand why is this behavior.

appreciate your help, thanks.

[chrislusf]

1. use latest version
2. use one filer or use a shared filer store. Seems there are some issue with connecting with filer peers.