Volume Vacuum Access is denied

[dianond]

Describe the bug

1. add file to weed
2. wait a moment
3. use curl -xdelete url
4. cmd display Access is denied.

System Setup
weed server -master.port=9304 -volume.port=9354 -dir="F:/temporary" -ip=192.168.2.223 -volume.max=60
Windows 2012 R2
weed version 1.56

Screenshots
when weed load
I0227 10:24:06 2100 file_util.go:20] Folder F:/temporary Permission: -rwxrwxrwx

I0227 10:24:06 2100 file_util.go:20] Folder F:/temporary Permission: -rwxrwxrwx

I0227 10:39:01 6120 volume_layout.go:238] Volume 3 becomes unwritable
I0227 10:39:01 6120 topology_vacuum.go:65] 0 Start vacuuming 3 on 192.168.2.223
:9354
I0227 10:39:01 6120 topology_vacuum.go:76] Complete vacuuming 3 on 192.168.2.22
3:9354
I0227 10:39:01 6120 topology_vacuum.go:95] Start Committing vacuum 3 on 192.168
.2.223:9354
I0227 10:39:01 6120 volume_vacuum.go:82] Committing volume 3 vacuuming...
E0227 10:39:01 6120 volume_grpc_vacuum.go:50] commit volume 3: rename F:/tempor
ary/3.cpd: rename F:/temporary/3.cpd F:/temporary/3.dat: Access is denied.
E0227 10:39:01 6120 topology_vacuum.go:103] Error when committing vacuum 3 on 1
92.168.2.223:9354: rpc error: code = Unknown desc = rename F:/temporary/3.cpd: r
ename F:/temporary/3.cpd F:/temporary/3.dat: Access is denied.

[chrislusf]

I have no windows machine to test on.

Can you try to run on F:/ and use "/temporary" directory instead?

[dianond]

thank you reply so quick

copy weed to F:/

i use bat

:loop
start /B /WAIT weed server -master.port=9304 -volume.port=9354 -dir="/temporary" -ip=192.168.2.223 -volume.max=60
timeout /t 5
goto loop

F:>start /B /WAIT weed server -master.port=9304 -volume.port=9354 -dir="/tempor
ary" -ip=192.168.2.223 -volume.max=60

I0227 13:20:00 4396 file_util.go:20] Folder /temporary Permission: -rwxrwxrwx
I0227 13:20:00 4396 file_util.go:20] Folder /temporary Permission: -rwxrwxrwx
E0227 13:20:00 4396 tls.go:49] load cert/key error: open : The system cannot fi
nd the file specified.
E0227 13:20:00 4396 tls.go:49] load cert/key error: open : The system cannot fi
nd the file specified.
I0227 13:20:00 4396 master_server.go:102] Volume Size Limit is 30000 MB
I0227 13:20:00 4396 disk_location.go:117] Store started on dir: /temporary with
0 volumes max 60
I0227 13:20:00 4396 disk_location.go:120] Store started on dir: /temporary with
0 ec shards
I0227 13:20:00 4396 master_server.go:188] adminScripts:
I0227 13:20:00 4396 volume_grpc_client_to_master.go:26] Volume server start wit
h seed master nodes: [192.168.2.223:9304]
I0227 13:20:00 4396 master.go:112] Start Seaweed Master 30GB 1.56 at 0.0.0.0:93
04
I0227 13:35:01 4396 volume_layout.go:238] Volume 6 becomes unwritable
I0227 13:35:01 4396 topology_vacuum.go:65] 0 Start vacuuming 6 on 192.168.2.223
:9354
I0227 13:35:01 4396 topology_vacuum.go:76] Complete vacuuming 6 on 192.168.2.22
3:9354
I0227 13:35:01 4396 topology_vacuum.go:95] Start Committing vacuum 6 on 192.168
.2.223:9354
I0227 13:35:01 4396 volume_vacuum.go:82] Committing volume 6 vacuuming...
E0227 13:35:01 4396 volume_grpc_vacuum.go:50] commit volume 6: rename /temporar
y/6.cpd: rename /temporary/6.cpd /temporary/6.dat: Access is denied.
E0227 13:35:01 4396 topology_vacuum.go:103] Error when committing vacuum 6 on 1
92.168.2.223:9354: rpc error: code = Unknown desc = rename /temporary/6.cpd: ren
ame /temporary/6.cpd /temporary/6.dat: Access is denied.
I0227 13:35:01 4396 volume_layout.go:238] Volume 7 becomes unwritable
I0227 13:35:01 4396 topology_vacuum.go:65] 0 Start vacuuming 7 on 192.168.2.223
:9354
I0227 13:35:01 4396 topology_vacuum.go:76] Complete vacuuming 7 on 192.168.2.22
3:9354
I0227 13:35:01 4396 topology_vacuum.go:95] Start Committing vacuum 7 on 192.168
.2.223:9354
I0227 13:35:01 4396 volume_vacuum.go:82] Committing volume 7 vacuuming...
E0227 13:35:01 4396 volume_grpc_vacuum.go:50] commit volume 7: rename /temporar
y/7.cpd: rename /temporary/7.cpd /temporary/7.dat: Access is denied.
E0227 13:35:01 4396 topology_vacuum.go:103] Error when committing vacuum 7 on 1
92.168.2.223:9354: rpc error: code = Unknown desc = rename /temporary/7.cpd: ren
ame /temporary/7.cpd /temporary/7.dat: Access is denied.
I0227 13:35:01 4396 volume_layout.go:238] Volume 1 becomes unwritable
I0227 13:35:01 4396 topology_vacuum.go:65] 0 Start vacuuming 1 on 192.168.2.223
:9354
I0227 13:35:01 4396 topology_vacuum.go:76] Complete vacuuming 1 on 192.168.2.22
3:9354
I0227 13:35:01 4396 topology_vacuum.go:95] Start Committing vacuum 1 on 192.168
.2.223:9354
I0227 13:35:01 4396 volume_vacuum.go:82] Committing volume 1 vacuuming...
E0227 13:35:01 4396 volume_grpc_vacuum.go:50] commit volume 1: rename /temporar
y/1.cpd: rename /temporary/1.cpd /temporary/1.dat: Access is denied.
E0227 13:35:01 4396 topology_vacuum.go:103] Error when committing vacuum 1 on 1
92.168.2.223:9354: rpc error: code = Unknown desc = rename /temporary/1.cpd: ren
ame /temporary/1.cpd /temporary/1.dat: Access is denied.

[dianond]

bat run with administrator

[dianond]

I0227 13:50:00 4396 topology_vacuum.go:65] 0 Start vacuuming 2 on 192.168.2.223
:9354
I0227 13:50:00 4396 topology_vacuum.go:76] Complete vacuuming 2 on 192.168.2.22
3:9354
I0227 13:50:00 4396 topology_vacuum.go:95] Start Committing vacuum 2 on 192.168
.2.223:9354
I0227 13:50:00 4396 volume_vacuum.go:82] Committing volume 2 vacuuming...
W0227 13:50:00 4396 needle_map_memory.go:69] sync file /temporary/2.idx failed,
sync /temporary/2.idx: file already closed
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x18 pc=0xd31ff9]

goroutine 2167 [running]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc000416c30
, 0x0, 0x0)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/storage/volu
me_vacuum.go:94 +0x1e9
github.com/chrislusf/seaweedfs/weed/storage.(*Store).CommitCompactVolume(0xc0000
22000, 0x2, 0xc000696000, 0x1de03c0)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/storage/stor
e_vacuum.go:26 +0x51
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).VacuumVolumeCommit(0x
c0000d5760, 0x2261f00, 0xc00076eab0, 0xc0007be924, 0xc0000d5760, 0xc00076eab0, 0
xc001719a80)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volum
e_grpc_vacuum.go:47 +0x59
github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb._VolumeServer_VacuumVolu
meCommit_Handler(0x1e58440, 0xc0000d5760, 0x2261f00, 0xc00076eab0, 0xc00061b0e0,
0x0, 0x2261f00, 0xc00076eab0, 0xc0007be920, 0x2)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/pb/volume_se
rver_pb/volume_server.pb.go:2917 +0x21e
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002ac580, 0x2280060, 0xc000
41cf00, 0xc0004c1700, 0xc000142e40, 0x329b048, 0x0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:995
+0x467
google.golang.org/grpc.(*Server).handleStream(0xc0002ac580, 0x2280060, 0xc00041c
f00, 0xc0004c1700, 0x0)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:127
5 +0xd9e
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0000942f0, 0xc0002ac580
, 0x2280060, 0xc00041cf00, 0xc0004c1700)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:710
+0xc2
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.23.0/server.go:708
+0xa8

[chrislusf]

Added a fix to delete the target file first then rename.

[dianond]

use 1.58 and test delete
log
I0302 17:17:23 5288 volume_layout.go:238] Volume 2 becomes unwritable
I0302 17:17:23 5288 topology_vacuum.go:65] 0 Start vacuuming 2 on 192.168.2.223:9354
I0302 17:17:24 5288 topology_vacuum.go:76] Complete vacuuming 2 on 192.168.2.223:9354
I0302 17:17:24 5288 topology_vacuum.go:95] Start Committing vacuum 2 on 192.168.2.223:9354
I0302 17:17:24 5288 volume_vacuum.go:83] Committing volume 2 vacuuming...
E0302 17:17:24 5288 volume_grpc_vacuum.go:50] commit volume 2: rename /temporary/2.cpd: rename /temporary/2.cpd /temporary/2.dat: Access is denied.
E0302 17:17:24 5288 topology_vacuum.go:103] Error when committing vacuum 2 on 192.168.2.223:9354: rpc error: code = Unknown desc = rename /temporary/2.cpd: rename /temporary/2.cpd /temporary/2.dat: Access is denied.

I0302 17:32:23 5288 topology_vacuum.go:65] 0 Start vacuuming 2 on 192.168.2.223:9354
I0302 17:32:23 5288 topology_vacuum.go:76] Complete vacuuming 2 on 192.168.2.223:9354
I0302 17:32:23 5288 topology_vacuum.go:95] Start Committing vacuum 2 on 192.168.2.223:9354
I0302 17:32:23 5288 volume_vacuum.go:83] Committing volume 2 vacuuming...
W0302 17:32:23 5288 needle_map_memory.go:69] sync file /temporary/2.idx failed, sync /temporary/2.idx: file already closed
I0302 17:32:23 5288 volume_vacuum.go:105] makeupDiff in CommitCompact volume 2 failed verifyIndexFileIntegrity /temporary/2.idx failed: invalid argument
I0302 17:32:23 5288 volume_loading.go:104] loading index /temporary/2.idx to memory
I0302 17:32:23 5288 topology_vacuum.go:106] Complete Committing vacuum 2 on 192.168.2.223:9354
I0302 17:32:23 5288 volume_layout.go:250] Volume 2 becomes writable

use ubuntu 1804 test is no problem

[chrislusf]

hi, this should be fixed now in release 1.61. Please help to confirm.