unexpected message when trying to delete locked object

[vkarak1]

When I am trying to delete locked object getting unexpected message:
rpc error: remove object via client: status: code = 1024 message = incomplete object PUT by placement: could not close object stream: (*putsvc.remoteTarget) could not put object to [/dns4/node1.neofs/tcp/8080]: write object via client: status: code = 2050 message = object is locked

When trying to rerun the command getting the different error message:
rpc error: remove object via client: status: code = 2052 message = object already removed

Expected Behavior

The following message should be displayed while the object is locked:
rpc error: remove object via client: status: code = 2050 message = object is locked

Steps to Reproduce (for bugs)

1. Create container
   neofs-cli container create --basic-acl public-read-write -r node1.neofs:8080 -w /etc/neofs/storage/wallet.json -p "REP 1"
2. allocate 500m object and put to the container:
   neofs-cli --rpc-endpoint node1.neofs:8080 -w /etc/neofs/storage/wallet.json object put --file /etc/neofs/storage/object1 --cid 32nNgCJCG6e3aZX178oWVYAJvpHiZ2XP8wkN6msrtujT
3. Issue container nodes command: neofs-cli container nodes --cid 32nNgCJCG6e3aZX178oWVYAJvpHiZ2XP8wkN6msrtujT -r node1.neofs:8080 -w /etc/neofs/storage/wallet.json
   Output:

Rep 1
        Node 1: 02df711d9f69027f4e299fa64fa87e984435d36eb5076146d9677ecd1d1d7cfe2f ONLINE /dns4/node4.neofs/tcp/8080
                Continent: Europe
                Country: Finland
                CountryCode: FI
                Deployed: YACZROKH
                Location: Helsinki (Helsingfors)
                Node: node4
                Price: 10
                SubDiv: Uusimaa
                SubDivCode: 18
                UN-LOCODE: FI HEL
        Node 2: 03de248780d29f80cb15418822ef206e6acae22c50d9428172376cbb9768594b37 ONLINE /dns4/node3.neofs/tcp/8080
                Continent: Europe
                Country: Sweden
                CountryCode: SE
                Deployed: YACZROKH
                Location: Stockholm
                Node: node3
                Price: 10
                SubDiv: Stockholms l�n
                SubDivCode: AB
                UN-LOCODE: SE STO
        Node 3: 02665dba76c7326948536ff98bdf936ebf7e8f7fe5f35622e40ace93ea48775836 ONLINE /dns4/node1.neofs/tcp/8080
                Continent: Europe
                Country: Russia
                CountryCode: RU
                Deployed: YACZROKH
                Location: Moskva
                Node: node1
                Price: 10
                SubDiv: Moskva
                SubDivCode: MOW
                UN-LOCODE: RU MOW

5. Issue object lock:

neofs-cli object lock -r node1.neofs:8080 -w /etc/neofs/storage/wallet.json --expire-at 38 32nNgCJCG6e3aZX178oWVYAJvpHiZ2XP8wkN6msrtujT EfTM8kj1JkFwJqvCsruo1DFKKMBBhg5RsW6HZSEf8sXq
    Lock object ID: CBZ5DUVJNsVN7qvBewNkoYT8gN9yfn8Fn3QfK8WMvoi1
    Objects successfully locked.

6. Try to issue "object delete" to node1 (got the expected result):

root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-cli --rpc-endpoint node1.neofs:8080 -w /etc/neofs/storage/wallet.json object delete --cid 32nNgCJCG6e3aZX178oWVYAJvpHiZ2XP8wkN6msrtujT --oid EfTM8kj1JkFwJqvCsruo1DFKKMBBhg5RsW6HZSEf8sXq
Enter password >
rpc error: remove object via client: status: code = 2050 message = object is locked

7. Try to issue "object delete" to node2 (fails with unexpected message):

root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-cli --rpc-endpoint node2.neofs:8080 -w /etc/neofs/storage/wallet.json object delete --cid 32nNgCJCG6e3aZX178oWVYAJvpHiZ2XP8wkN6msrtujT --oid EfTM8kj1JkFwJqvCsruo1DFKKMBBhg5RsW6HZSEf8sXq
Enter password >
rpc error: remove object via client: status: code = 1024 message = incomplete object PUT by placement: could not close object stream: (*putsvc.remoteTarget) could not put object to [/dns4/node3.neofs/tcp/8080]: write object via client: status: code = 2050 message = object is locked

8. Try to issue one more time the command in step7:

root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-cli --rpc-endpoint node2.neofs:8080 -w /etc/neofs/storage/wallet.json object delete --cid 32nNgCJCG6e3aZX178oWVYAJvpHiZ2XP8wkN6msrtujT --oid EfTM8kj1JkFwJqvCsruo1DFKKMBBhg5RsW6HZSEf8sXq
Enter password >
rpc error: remove object via client: status: code = 2052 message = object already removed

9. Try to issue "object delete" to node3 / node4 (got the expected results):

root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-cli --rpc-endpoint node3.neofs:8080 -w /etc/neofs/storage/wallet.json object delete --cid 32nNgCJCG6e3aZX178oWVYAJvpHiZ2XP8wkN6msrtujT --oid EfTM8kj1JkFwJqvCsruo1DFKKMBBhg5RsW6HZSEf8sXq
Enter password >
rpc error: remove object via client: status: code = 2050 message = object is locked
root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-cli --rpc-endpoint node4.neofs:8080 -w /etc/neofs/storage/wallet.json object delete --cid 32nNgCJCG6e3aZX178oWVYAJvpHiZ2XP8wkN6msrtujT --oid EfTM8kj1JkFwJqvCsruo1DFKKMBBhg5RsW6HZSEf8sXq
Enter password >
rpc error: remove object via client: status: code = 2050 message = object is locked

Attached logs

Your Environment

NeoFS Storage node
Version: v0.33.0-31-ge1be0180
GoVersion: go1.18.4

[vkarak1]

The issue can be reproduced only with REP1, I was not able to reproduce with REP2.

[carpawell]

Could be related to the fact that we do not broadcast LOCK objects as TOMBSTONES. Also, that could be the same reason resyncing could not work correctly.

[fyrchik]

Closed via #2008.