XFormsReplicationTest occasionally getting stuck

[avernet]

• This is happening both on Travis and locally.
• To reproduce locally, run orbeonWarJS / testOnly *XFormsReplicationTest.
• When it gets stuck, it is always in the same place:
  • after shutting down B, in updateWindowsAndAssert,
  • when doing the updateWindow on window B.
• The request reaches the Tomcat A, since we've shut down B.
• When this fails
  • We get the following log (enabled dev logging): TomcatA-failure-log.txt.
  • We see the request for quantity = 50.
  • We are restoring the dynamic state, but then throw an "Got unexpected request sequence number".

[avernet]

• The issue still happens occasionally after uncommenting the await(delay(5.seconds)) we had before shutting down B.
  • In orbeon-war/js/src/test/resources/resources/config/ehcache.xml, we set the property replicateAsynchronously=false.
  • The issue still happens in about 50% of the runs after increasing the delay to 20 seconds, which makes me think that the issue is unrelated to this.

[avernet]

• Log for success case: TomcatA-success-log.txt.

[ebruchez]

Curious: I can't see the code that updates the dynamic state after every request when replication is enabled. However, this works in some cases!

• find code that updates the dynamic state after every request
• check if somehow we might not store the newer sequence number associated with the latest dynamic state

[ebruchez]

We can do this locally, as it appears to happen about 50% of the time.