Occasional failure TccLoadBalanceSenderTest.participateFailedThenRetry

[coolbeevip]

alpha server 0.4.0 master

Use mvn clean install -Pdemo,spring-boot-2

error log

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.servicecomb.pack.omega.connector.grpc.tcc.GrpcTccClientMessageSenderTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.594 sec - in org.apache.servicecomb.pack.omega.connector.grpc.tcc.GrpcTccClientMessageSenderTest
Running org.apache.servicecomb.pack.omega.connector.grpc.tcc.TccLoadBalanceSenderTest
Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.391 sec <<< FAILURE! - in org.apache.servicecomb.pack.omega.connector.grpc.tcc.TccLoadBalanceSenderTest
participateFailedThenRetry(org.apache.servicecomb.pack.omega.connector.grpc.tcc.TccLoadBalanceSenderTest)  Time elapsed: 2.388 sec  <<< FAILURE!
java.lang.AssertionError: 

Expected: is <3>
     but: was <4>
        at org.apache.servicecomb.pack.omega.connector.grpc.tcc.TccLoadBalanceSenderTest.participateFailedThenRetry(TccLoadBalanceSenderTest.java:213)

Running org.apache.servicecomb.pack.omega.connector.grpc.LoadBalanceContextBuilderTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec - in org.apache.servicecomb.pack.omega.connector.grpc.LoadBalanceContextBuilderTest
Running org.apache.servicecomb.pack.omega.connector.grpc.PushBackReconnectRunnableTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.025 sec - in org.apache.servicecomb.pack.omega.connector.grpc.PushBackReconnectRunnableTest
Running org.apache.servicecomb.pack.omega.connector.grpc.saga.RetryableMessageSenderTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec - in org.apache.servicecomb.pack.omega.connector.grpc.saga.RetryableMessageSenderTest
Running org.apache.servicecomb.pack.omega.connector.grpc.saga.SagaLoadBalanceSenderWithTLSTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.579 sec - in org.apache.servicecomb.pack.omega.connector.grpc.saga.SagaLoadBalanceSenderWithTLSTest
Running org.apache.servicecomb.pack.omega.connector.grpc.saga.SagaLoadBalancedSenderTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.497 sec - in org.apache.servicecomb.pack.omega.connector.grpc.saga.SagaLoadBalancedSenderTest

Results :

Failed tests: 
  TccLoadBalanceSenderTest.participateFailedThenRetry:213 
Expected: is <3>
     but: was <4>

I found a possible problem in TccLoadBalanceSender.participationStart

  @Override
  public AlphaResponse participationStart(ParticipationStartedEvent participationStartedEvent) {
    do {
      final TccMessageSender messageSender = pickMessageSender();
      Optional<AlphaResponse> response = doGrpcSend(messageSender, participationStartedEvent, new SenderExecutor<ParticipationStartedEvent>() {
        @Override
        public AlphaResponse apply(ParticipationStartedEvent event) {
          return messageSender.participationStart(event);
        }
      });
      if (response.isPresent()) return response.get();
    } while (!Thread.currentThread().isInterrupted());

    throw new OmegaException("Failed to send event " + participationStartedEvent + " due to interruption");
  }

if response is empty it will be executed cyclically. May be this reason?

[WillemJiang]

do... while is intended. The sender keeps sending the message until there is no exception.
As we just wait for at most 2 seconds for the restarting of the service. It could cause some trouble if server doesn't start in 2 seconds.
I just updated the wait time to 3 seconds to make sure the server is back for business on a slower box.