Storage group wasn't deleted after lifetime was expired #2016

anikeev-yadro · 2022-11-03T11:06:04Z

Expected Behavior

Storage group should be deleted after lifetime expired

Current Behavior

Storage group wasn't deleted after lifetime was expired

Steps to Reproduce (for bugs)

Create storagegroup with lifetime=1

COMMAND: neofs-cli --config /jenkins/workspace/sbercloud_test/tmp.nzkHtw9gNy/neofs-testcases/wallet_config.yml storagegroup put --rpc-endpoint '172.26.160.37:8080' --wallet '/jenkins/workspace/sbercloud_test/tmp.nzkHtw9gNy/neofs-testcases/TemporaryDir/9a64eee7-52c1-4ee2-848e-11fc954f6550.json' --cid '7QwoRQ8cW63GRyT5DZJbXcpkBhyaCHsPsUBvV6PGQAHt' --members '3nDYw6tVNE15HsqgJHcfk8uLHHLfeLoysp6kZmysUxii' --lifetime 1
RETCODE: 0

STDOUT:
Storage group successfully stored
  ID: HRCKtbDHaVXcb51F3vde89fFw99E9VfZXoWd8zdTeGCu
  CID: 7QwoRQ8cW63GRyT5DZJbXcpkBhyaCHsPsUBvV6PGQAHt

STDERR:

Start / End / Elapsed	 08:10:27.037715 / 08:10:27.474764 / 0:00:00.437049

Tick epoch 2 times

COMMAND: neofs-adm --config /jenkins/workspace/sbercloud_test/tmp.nzkHtw9gNy/tatlin-object-testsetup/.setup/config.yaml morph force-new-epoch 
RETCODE: 0

STDOUT:
Current epoch: 100, increase to 101.
Waiting for transactions to persist...

STDERR:
Start / End / Elapsed	 08:10:27.476317 / 08:10:39.208320 / 0:00:11.732003

COMMAND: neofs-adm --config /jenkins/workspace/sbercloud_test/tmp.nzkHtw9gNy/tatlin-object-testsetup/.setup/config.yaml morph force-new-epoch 
RETCODE: 0

STDOUT:
Current epoch: 101, increase to 102.
Waiting for transactions to persist...

STDERR:
Start / End / Elapsed	 08:10:39.209956 / 08:10:46.896181 / 0:00:07.686225

Try to get storage group

COMMAND: neofs-cli --config /jenkins/workspace/sbercloud_test/tmp.nzkHtw9gNy/neofs-testcases/wallet_config.yml storagegroup get --rpc-endpoint '172.26.160.37:8080' --wallet '/jenkins/workspace/sbercloud_test/tmp.nzkHtw9gNy/neofs-testcases/TemporaryDir/9a64eee7-52c1-4ee2-848e-11fc954f6550.json' --cid '7QwoRQ8cW63GRyT5DZJbXcpkBhyaCHsPsUBvV6PGQAHt' --id 'HRCKtbDHaVXcb51F3vde89fFw99E9VfZXoWd8zdTeGCu'
RETCODE: 0

STDOUT:
Expiration epoch: 101
Group size: 1000
Group hash: <empty>
Members:
	3nDYw6tVNE15HsqgJHcfk8uLHHLfeLoysp6kZmysUxii

STDERR:
Start / End / Elapsed	 08:10:46.897948 / 08:10:47.281773 / 0:00:00.383825

Logs:d0bed81bfd1c6fc4.zip
Versions:

NeoFS node 0.33.0-78-g846ff515

Your Environment

Server setup and configuration:
cloud, 4 VMs, 4 SN, 4 http qw, 4 s3 gw

Operating System and version (uname -a):
linux vedi 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

anikeev-yadro · 2022-11-10T07:45:04Z

This bug always reproduce on virtual environment, but not reproduce on devenv.

anikeev-yadro · 2022-12-22T07:05:16Z

This is the automatic test testsuites.acl.storage_group.test_storagegroup.TestStorageGroup#test_storagegroup_lifetime
It was reproduced today:
Test log:

03:27:13 [INFO] Init new wallet: /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/f826baaa-03f5-40e4-96b3-b619359109a2.json, address: NiQn5eBZ5sTHCDntqrUagWobmF7HZ2gaHs
03:27:13 [INFO] Executing command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml container create --rpc-endpoint '172.26.160.46:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json' --await --policy 'REP 2 IN X CBF 1 SELECT 4 FROM * AS X'
03:27:23 [INFO] Command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml container create --rpc-endpoint '172.26.160.46:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json' --await --policy 'REP 2 IN X CBF 1 SELECT 4 FROM * AS X'
Success:
return code: 0 
Output: container ID: F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn
awaiting...
container has been persisted on sidechain

03:27:23 [INFO] Container created; waiting until it is persisted in the sidechain
03:27:23 [INFO] Executing command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml container list --rpc-endpoint '172.26.160.46:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json'
03:27:23 [INFO] Command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml container list --rpc-endpoint '172.26.160.46:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json'
Success:
return code: 0 
Output: 2KUuzx1cgVJBMkxTuTuaApU1QBqpvvbpkk2Y2Hz96jRq
6QfLwCfmkx1L612HYZXwhD5XUv3C1eQESeJrqJDLKf4o
8fmn4DRn9N4aShJgAFkZPKJMPyLnsBW1dKvSv7snBTNs
8rodTA2UapFr949a42ToizEV7HdtJsHHurxqNikqnU1A
9KcFcKKWtAjZzkDPWRJVqch786LKTMCkCRVyyzNLQUAc
9UxaUtJu6Q8f1qfm52fqv9QjBHueycY24jBuZGF8zLXG
A2r9vXia19LASibTVsmadtKxHXgr6WGWZ9tXZ97WQjAu
ApBVBbj44gAJTkSFciJ5oWUjHcQoJTdWPmgHMmMCHxQ7
BCXTwDgaXu7zR54mXKNv61zwg8WWABtPpaJTDw9yATSa
CKjUnuDiB7osi2nQ6VGTE6pHBbXY8v2kWkEbsAxP8sqk
DWJTrARxnZh5ipAiJuAgg7nNk4o9fFS3tiAmsfvcqtmo
DhTFngd2pe1hMhZBrJrVmhNgRADxSB2CKisebBGPExNJ
Dn9oW5LCMx7M5fPfH4oCECSTQNY3EXaBa7FnDFpNoPkK
ExJ2KbHWDkLzwixrcFHvDUc2eoatp7kcdqTCYdoZvpoq
F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn
FE9ErUSfD1R4YPHqaE3WE7Ri1AHQ1RAuPCMHUb8c5Vi7
Fo4opK5hPFys4F6cXEEuGLF7igVztUYQ9vVPi7RTuaoJ
GiBn6S6vzBLiB11A2fpTMZb8LKRmp5Nh8gqVQtwgDJF6
HCFEMbMUnzUMa9ZV4ekTW5FyzRTA49p5GApjLpEESMYh
HgM1MvVUtKUe1iYatX8hFpfTiXRWRpxzPRaNo5B4B1oj

03:27:23 [INFO] Containers: 
CommandResult(stdout='2KUuzx1cgVJBMkxTuTuaApU1QBqpvvbpkk2Y2Hz96jRq\n6QfLwCfmkx1L612HYZXwhD5XUv3C1eQESeJrqJDLKf4o\n8fmn4DRn9N4aShJgAFkZPKJMPyLnsBW1dKvSv7snBTNs\n8rodTA2UapFr949a42ToizEV7HdtJsHHurxqNikqnU1A\n9KcFcKKWtAjZzkDPWRJVqch786LKTMCkCRVyyzNLQUAc\n9UxaUtJu6Q8f1qfm52fqv9QjBHueycY24jBuZGF8zLXG\nA2r9vXia19LASibTVsmadtKxHXgr6WGWZ9tXZ97WQjAu\nApBVBbj44gAJTkSFciJ5oWUjHcQoJTdWPmgHMmMCHxQ7\nBCXTwDgaXu7zR54mXKNv61zwg8WWABtPpaJTDw9yATSa\nCKjUnuDiB7osi2nQ6VGTE6pHBbXY8v2kWkEbsAxP8sqk\nDWJTrARxnZh5ipAiJuAgg7nNk4o9fFS3tiAmsfvcqtmo\nDhTFngd2pe1hMhZBrJrVmhNgRADxSB2CKisebBGPExNJ\nDn9oW5LCMx7M5fPfH4oCECSTQNY3EXaBa7FnDFpNoPkK\nExJ2KbHWDkLzwixrcFHvDUc2eoatp7kcdqTCYdoZvpoq\nF8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn\nFE9ErUSfD1R4YPHqaE3WE7Ri1AHQ1RAuPCMHUb8c5Vi7\nFo4opK5hPFys4F6cXEEuGLF7igVztUYQ9vVPi7RTuaoJ\nGiBn6S6vzBLiB11A2fpTMZb8LKRmp5Nh8gqVQtwgDJF6\nHCFEMbMUnzUMa9ZV4ekTW5FyzRTA49p5GApjLpEESMYh\nHgM1MvVUtKUe1iYatX8hFpfTiXRWRpxzPRaNo5B4B1oj\n', stderr='', return_code=0)
03:27:23 [INFO] File with size 1000 bytes has been generated: /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/3fab2dd1-e17f-4d45-9113-1fe42bb1927b
03:27:23 [INFO] Executing command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml object put --rpc-endpoint '172.26.160.126:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json' --cid 'F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn' --file '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/3fab2dd1-e17f-4d45-9113-1fe42bb1927b' --no-progress
03:27:24 [INFO] Command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml object put --rpc-endpoint '172.26.160.126:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json' --cid 'F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn' --file '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/3fab2dd1-e17f-4d45-9113-1fe42bb1927b' --no-progress
Success:
return code: 0 
Output: [/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/3fab2dd1-e17f-4d45-9113-1fe42bb1927b] Object successfully stored
  OID: CwZw71JwqGBJVvXo4VLybCWuDb2xHZYFuvdtRRqMHiEs
  CID: F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn

03:27:24 [INFO] Executing command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml storagegroup put --rpc-endpoint '172.26.160.46:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json' --cid 'F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn' --members 'CwZw71JwqGBJVvXo4VLybCWuDb2xHZYFuvdtRRqMHiEs' --lifetime 1
03:27:24 [INFO] Command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml storagegroup put --rpc-endpoint '172.26.160.46:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json' --cid 'F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn' --members 'CwZw71JwqGBJVvXo4VLybCWuDb2xHZYFuvdtRRqMHiEs' --lifetime 1
Success:
return code: 0 
Output: Storage group successfully stored
  ID: G7r3jRX8XdvFWddP2Ww7VYKc6J8ZUVWYVd7MAbr2RYxT
  CID: F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn

03:27:24 [INFO] Executing command: neofs-adm --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/tatlin-object-testsetup/.setup/config.yaml morph force-new-epoch 
03:27:30 [INFO] Command: neofs-adm --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/tatlin-object-testsetup/.setup/config.yaml morph force-new-epoch 
Success:
return code: 0 
Output: Contract group wallet is missing, initialize at /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/tatlin-object-testsetup/.setup/alphabet-wallets/contract.json
Current epoch: 5, increase to 6.
Waiting for transactions to persist...

03:27:30 [INFO] Executing command: neofs-adm --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/tatlin-object-testsetup/.setup/config.yaml morph force-new-epoch 
03:27:38 [INFO] Command: neofs-adm --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/tatlin-object-testsetup/.setup/config.yaml morph force-new-epoch 
Success:
return code: 0 
Output: Current epoch: 6, increase to 7.
Waiting for transactions to persist...

03:27:38 [INFO] Executing command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml storagegroup get --rpc-endpoint '172.26.160.46:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json' --cid 'F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn' --id 'G7r3jRX8XdvFWddP2Ww7VYKc6J8ZUVWYVd7MAbr2RYxT'
03:27:38 [INFO] Command: neofs-cli --config /jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/wallet_config.yml storagegroup get --rpc-endpoint '172.26.160.46:8080' --wallet '/jenkins/workspace/sbercloud_functional_tests/tmp.qof9qToL6Z/neofs-testcases/TemporaryDir/ea754405-ec42-463d-8e0d-3a9c97d72ee1.json' --cid 'F8iiiTpQcQf9cULFUTCUZUssmRXVUAiqB9PxaVG59KCn' --id 'G7r3jRX8XdvFWddP2Ww7VYKc6J8ZUVWYVd7MAbr2RYxT'
Success:
return code: 0 
Output: Expiration epoch: 6
Group size: 1000
Group hash: <empty>
Members:
	CwZw71JwqGBJVvXo4VLybCWuDb2xHZYFuvdtRRqMHiEs

Nodes logs: 51cb9b2da4a56a59.zip
Version: 0.34.0-126-gcde6b893-dirty

carpawell · 2023-01-10T18:28:42Z

@anikeev-yadro, as i understand, it is hard-to-reproduce bug and happens really rare. The issue was closed because of another problem (that could be easily reproduced manually (with dev's slow hands)). I tried to reproduce it locally 20 times (via neofs-testcases of course) and did not succeed. Forcing a new epoch is quite a complex process: consensus b/w neo-go nodes, subscription receiving by SN and updating internal state, while the test is written to force a new epoch and request SG immediately. Can we try to sleep for 1 second before requesting SG? /cc @abereziny

abereziny · 2023-01-16T12:59:28Z

Since epoch change process related to network/events/notifications models we must wait for epochs to be processed.
Test should be fixed by #502 and #503.
I recommend to add note to the product documentation about epochs are not changed immediatelly.

carpawell · 2023-01-19T07:45:44Z

@anikeev-yadro, lets close it if it does not reproduce for e.g. next 10 days, and just reopen it if it appears one more time?

roman-khimov · 2023-08-25T15:58:20Z

A test issue, not relevant now.

carpawell self-assigned this Nov 14, 2022

carpawell mentioned this issue Dec 20, 2022

Fix/Storage Group expiration #2167

Merged

roman-khimov removed the community label Mar 21, 2023

roman-khimov removed the yaczrokh label Apr 22, 2023

roman-khimov unassigned carpawell Apr 23, 2023

roman-khimov closed this as not planned Won't fix, can't repro, duplicate, stale Aug 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage group wasn't deleted after lifetime was expired #2016

Storage group wasn't deleted after lifetime was expired #2016

anikeev-yadro commented Nov 3, 2022

anikeev-yadro commented Nov 10, 2022

anikeev-yadro commented Dec 22, 2022

carpawell commented Jan 10, 2023

abereziny commented Jan 16, 2023 •

edited

Loading

carpawell commented Jan 19, 2023

roman-khimov commented Aug 25, 2023

Storage group wasn't deleted after lifetime was expired #2016

Storage group wasn't deleted after lifetime was expired #2016

Comments

anikeev-yadro commented Nov 3, 2022

Expected Behavior

Current Behavior

Steps to Reproduce (for bugs)

Your Environment

anikeev-yadro commented Nov 10, 2022

anikeev-yadro commented Dec 22, 2022

carpawell commented Jan 10, 2023

abereziny commented Jan 16, 2023 • edited Loading

carpawell commented Jan 19, 2023

roman-khimov commented Aug 25, 2023

abereziny commented Jan 16, 2023 •

edited

Loading