unable to force-new-epoch

[vkarak1]

The node1 left in epoch# 71 due to internal addresses of node2, node3, and node4 had been rejected about 1 day.

route add 172.26.162.244 reject; route add 172.26.162.70 reject; route add 172.26.162.119 reject; 

Then I deleted nodes from rejection list:

route del 172.26.162.244 reject; route del 172.26.162.70 reject; route del 172.26.162.119 reject;

And issued force-new-epoch and epoch remains the same on node1 but was successfuly forced to the next epoch on node3:

root@az:/home/service# neofs-cli netmap snapshot -g -r inode1:8080
Epoch: 71
............ etc
root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-adm morph force-new-epoch -c configuration/config.yaml
Current epoch: 124, increase to 125.
Waiting for transactions to persist...
root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-cli netmap snapshot -g -r inode1:8080
Epoch: 71
............ etc
root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-adm morph force-new-epoch -c configuration/config.yaml
Current epoch: 125, increase to 126.
Waiting for transactions to persist...
root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-cli netmap snapshot -g -r inode1:8080
Epoch: 71
............ etc
root@az:/etc/neofs/storage/tatlin-object-sber-tfstate/vkarakozov# neofs-cli netmap epoch -g -r inode1:8080
71

Please find the output of node3 in after force-new-epoch command has been sent:

root@vedi:/home/service# neofs-cli netmap snapshot -g -r inode3:8080
Epoch: 126
root@vedi:/home/service# neofs-cli netmap epoch -g -r inode3:8080
126

After 10-20 minutes it looks like that node1 synchronized with others and had corresponding epoch and I was not able to reproduce this one more time.

Expected Behavior

neofs-adm morph force-new-epoch cmd have to force a new epoch for all available nodes.

force-new-epoch.zip

Your Environment

Linux az 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux

NeoFS Storage node
Version: v0.34.0-120-g3a3b47b9-dirty
GoVersion: go1.18.4

[carpawell]

Seems like OK situation: neo-go was cut off from its friends like the neofs-storage too. Alone node did not receive blocks and became outdated. After breaking (connection) blocks, it started sync process and was not able to provide the actual epoch number to its subscribers.

After 10-20 minutes it looks like that node1 synchronized with others and had corresponding epoch and I was not able to reproduce this one more time.

That seems like a sync process finish.

[carpawell]

But on the other hand Current epoch: 124, increase to 125. says that neo-go already in sync and thought that its epoch is not 71.

The issue is missing logs for sure, hard to say what was happening having only IR's info logs.

[roman-khimov]

It's about chain synchronization effectively.