HDDS-5249. Race Condition between Full and Incremental Container Reports

[sodonnel]

What changes were proposed in this pull request?

During testing we came across an issue with ICR and FCR handing.

The following log shows the issue:

2021-05-18 13:14:15,394 DEBUG org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Processing replica of container #1 from datanode 945aa180-5cff-4298-a8ad-8197542e4562{ip: 172.27.108.136, host: quasar-nqdywv-7.quasar-nqdywv.root.hwx.site, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}


2021-05-18 13:14:15,394 DEBUG org.apache.hadoop.hdds.scm.container.IncrementalContainerReportHandler: Processing replica of container #1001 from datanode 945aa180-5cff-4298-a8ad-8197542e4562{ip: 172.27.108.136, host: quasar-nqdywv-7.quasar-nqdywv.root.hwx.site, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}


2021-05-18 13:14:15,394 DEBUG org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Processing replica of container #2 from datanode 945aa180-5cff-4298-a8ad-8197542e4562{ip: 172.27.108.136, host: quasar-nqdywv-7.quasar-nqdywv.root.hwx.site, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}
2021-05-18 13:14:15,394 DEBUG org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Processing replica of container #3 from datanode 945aa180-5cff-4298-a8ad-8197542e4562{ip: 172.27.108.136, host: quasar-nqdywv-7.quasar-nqdywv.root.hwx.site, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}
2021-05-18 13:14:15,394 DEBUG org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Processing replica of container #4 from datanode 945aa180-5cff-4298-a8ad-8197542e4562{ip: 172.27.108.136, host: quasar-nqdywv-7.quasar-nqdywv.root.hwx.site, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}
...

In the above log, SCM is processing both an ICR and FCR for the same Datanode at the same time. The FCR does not container container #1001.

The FCR starts first, and it takes a snapshot of the containers on the node via NodeManager.

Then it starts processing the containers one by one.

The ICR then starts, and it added #1001 to the ContainerManager and to the NodeManager.

When the FCR completes, it replaces the list of containers in NodeManager with those in the FCR.

At this point, container #1001 is in the ContainerManager, but it is not listed against the node in NodeManager.

This would get fixed by the next FCR, but then the node goes dead. The dead node handler runs and uses the list of containers in NodeManager to remove all containers for the node. As #1001 is not listed, it is not removed by the DeadNodeManager. This means the container will never been seen as under replicated, as 3 copies will exist forever in the ContainerManager.

This issue is quite tricky to fully fix. There are two issues:

1. Parallel processing of ICR and FCR can lead to data inconsistency between the ComtainerManager and NodeManager. This is what caused the bug above.

2. A FCR wiping out a reference to a container recently sent in an ICR, but which is not included in the FCR.

The second issue is less serious, as the next FCR will fix the problem, as the FCRs are produced approximately every 60 seconds by default.

We can fix problem 1 quite easily by synchronising on the datanode when processing FCRs and ICRs, that will ensure the data inconsistency will not happen.

This PR is for issue 1, and we should probably create a followup issue for 2.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5249

How was this patch tested?

Added a new test to reproduce the race condition and verified it passes after the code change.

[GlenGeng-awx]

cc @GlenGeng

[GlenGeng-awx]

+1. Thanks for the fix of this subtle issue.

BTW, about the issue 2,

A FCR wiping out a reference to a container recently sent in an ICR, but which is not included in the FCR.

Right now the order of FCR and ICR is not guaranteed, we may add some monotonically increasing
sequence number in datanode side so that SCM can figure out a stale FCR report and drop it.

[sodonnel]

@GlenGeng Thanks for the review. I will go ahead and commit this one, and raise a new Jira for issue 2.

I think you are correct, in that we may need a monotonically increasing sequence number in the datanode, plus maybe some locking in the DN to guarantee the contents of the FCR.

The hard part, is figuring out what to do with that information on SCM. I wonder if we need to attach the "report sequence number" to every replica in SCM. Then we can say:

• The FCR is has sequence 100, and it does not have container 1001.
• 1001 is in SCM, but added by ICR sequence 101.
• Right now, the FCR processing would remove container 1001, but with the sequence we could ensure it does not remove anything created by a newer sequence number.

However this approach adds a new field to every replica which would have a memory overhead, so we need to try to think of a better way.

[sodonnel]

HDDS-5267 raised to track / fix the second issue mentioned here.