found and fixed a bug in raft integration tests

[pfi79]

Often fails this test

• https://github.com/hyperledger/fabric/actions/runs/7493668256/job/20399967195
• https://github.com/hyperledger/fabric/actions/runs/7483402876/job/20368656381
• https://github.com/hyperledger/fabric/actions/runs/7462441029/job/20304890927

Failure occurs when waiting for the orderer to delete a channel

Here is an explanation: why they added the removal of the channel with orderer

I'll describe what's going on:

• the orderer is in a constant cycle periodically performing the function
• send a command to remove channel. It passes the following functions serveRemove, RemoveChannel (r.lock monopole is taken in this function), Halt (the actual removal of the channel is expected here)
• let's return to the main loop, this is where the actual deletion should take place, it wants to execute the function SwitchFollowerToChain, but she can't, she needs a r.lock monopole.

==================================

The error in the raft tests is due to the struggle for monopoly control of r.lock in the SwitchChainToFollower and RemoveChannel functions.

RemoveChannel grabs monopoly control and waits for the channel to terminate. During termination, SwitchChainToFollower tries to grab monopoly control and fails.

Here is my variant of solving the error. If someone suggests a more correct and reddish way, it would be great.

[yacovm]

can you perhaps write a test that reproduces this problem?

[yacovm]

@tock-ibm do you want to take a look at this?

[pfi79]

can you perhaps write a test that reproduces this problem?

I was wondering how to do it in a manageable way. I couldn't do it with integration tests. It's a complete asynchronous interaction.
Ideally, I need to suspend the peer in a certain place, send a command to remove the channel and continue the peer.

Here is a test test often crashes. For the last 2-3 months all errors in raft integration tests are the same error.

For example:

https://github.com/hyperledger/fabric/actions/runs/7493668256/job/20399967195
https://github.com/hyperledger/fabric/actions/runs/7483402876/job/20368656381
https://github.com/hyperledger/fabric/actions/runs/7462441029/job/20304890927

[pfi79]

My solution works, but it's not pretty, I don't like it. But it fixes the error.

[pfi79]

Here is an example of logs with normal operation

[e][OrdererOrg.orderer1] 2024-01-11 19:50:21.851 UTC 00ac INFO [orderer.common.follower] pullAfterJoin -> Pulling after join channel=testchannel
[e][OrdererOrg.orderer1] 2024-01-11 19:50:21.855 UTC 00ad INFO [orderer.common.follower] pullAfterJoin -> Pulled after join channel=testchannel
[e][OrdererOrg.orderer1] 2024-01-11 19:50:21.855 UTC 00ae INFO [orderer.common.follower] pull -> Block pulling finished successfully, going to switch from follower to a consensus.Chain channel=testchannel
[e][OrdererOrg.orderer1] 2024-01-11 19:50:21.855 UTC 00af INFO [orderer.common.follower] halt -> Stopped channel=testchannel

Here are the logs at the time of the error. After that orderer1 does nothing else. Channel removal is in progress (halt -> Stopped)

[e][OrdererOrg.orderer1] 2024-01-11 19:50:22.351 UTC 00ca INFO [orderer.common.follower] pullAfterJoin -> Pulling after join channel=testchannel
STEP: Removing channel from the first orderer 01/11/24 19:50:22.352
[e][OrdererOrg.orderer1] 2024-01-11 19:50:22.354 UTC 00cb INFO [orderer.common.follower] halt -> Stopped channel=testchannel
[e][OrdererOrg.orderer1] 2024-01-11 19:50:22.356 UTC 00cc INFO [orderer.common.follower] pullAfterJoin -> Pulled after join channel=testchannel
[e][OrdererOrg.orderer1] 2024-01-11 19:50:22.356 UTC 00cd INFO [orderer.common.follower] pull -> Block pulling finished successfully, going to switch from follower to a consensus.Chain channel=testchannel