HDDS-3179 Pipeline placement based on Topology does not have fallback

[timmylicheng]

What changes were proposed in this pull request?

When rack awareness and topology is enabled, pipeline placement can fail when there is only one node on the rack.
Should add fall back logic to search for nodes from other racks.
(Please fill in changes proposed in this fix)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3179

(Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)

Please replace this section with the link to the Apache JIRA)

How was this patch tested?

UT
(Please explain how this patch was tested. Ex: unit tests, manual tests)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)

[sodonnel]

Acceptance is failing with:

 ==============================================================================
Execute PI calculation                                                | FAIL |
1 != 0
------------------------------------------------------------------------------
Execute WordCount                                                     | FAIL |
1 != 0
------------------------------------------------------------------------------
ozonesecure-mr-mapreduce :: Execute MR jobs                           | FAIL |

This is known issue and will be fixed soon - HDDS-3284.

Integration tests are flaky. it-Freon:

[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 395.168 s <<< FAILURE! - in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator
[ERROR] bigFileThan2GB(org.apache.hadoop.ozone.freon.TestRandomKeyGenerator)  Time elapsed: 326.297 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)

One of these issues covers it: https://issues.apache.org/jira/browse/HDDS-3266 and it-freon: https://issues.apache.org/jira/browse/HDDS-3257

it-client I am not sure

[sodonnel]

Thanks for the updates here. I think the code looks much cleaner now with the debug statements and refactored block in getResultSet().

There are just a couple of minor changes needed to finish this one off.

[timmylicheng]

Thanks for the updates here. I think the code looks much cleaner now with the debug statements and refactored block in getResultSet().

There are just a couple of minor changes needed to finish this one off.

Thanks for the detailed review. It really helps. @sodonnel

[sodonnel]

Thanks for quickly addressing the final issues.

I am +1 on this now. I will commit it later pending the CI checks looking good.

[sodonnel]

All tests are green except this one, which I have seen fail in several other PRs, so it is flaky:

 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.235 s - in org.apache.hadoop.ozone.freon.TestDataValidateWithUnsafeByteOperations
[INFO] Running org.apache.hadoop.ozone.freon.TestRandomKeyGenerator
[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 338.009 s <<< FAILURE! - in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator
[ERROR] bigFileThan2GB(org.apache.hadoop.ozone.freon.TestRandomKeyGenerator)  Time elapsed: 276.915 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)

[elek]

If you see a flaky test, please disable it (without and issue) + create a new open issue and repeat the build. There is some risk that one flakiness hides a other. Merging patches without green build makes harder to debug flaky tests.