Fix flaky test caused by .toArray()#403
Conversation
|
Hi reviewers, is this fix satisfactory? If no, can you please suggest a better approach if you have it in mind? |
|
Hi, thank you for digging deeply into our tests. This test case actually doesn't measure the order of the set, we don't expect that. The test is just making sure the map field is initialized. |
|
Yes, it intentionally mixes the order to produce unpredictability which helps detect tests that can non-deterministically pass and fail when run on the same version of code. |
|
OK. If you want to fix this, you could change to codes to compare the two HashSets. We don't require the order actually. |
|
Thanks for your feedback! I will make the necessary changes and update the pull request. |
|
Hello, I have updated the fix to just compare the two HashSets without caring about the order. Can you please review the change? |
|
Thanks. Merged. |
Description
Flaky Test found using NonDex by running the command -
mvn -pl apm-commons/apm-util edu.illinois:nondex-maven-plugin:1.1.2:nondex -Dtest=org.apache.skywalking.apm.util.ConfigInitializerTest#testInitializeThe logged failure -
Investigation
The test fails at
ConfigInitializerTest.testInitialize:70with the error regarding a mismatch of elements of the expected array of Strings with the actual elements from theTestPropertiesObject.Level1Object.SET_STR_ATTR.toArray()on line 70. The SET_STR_ATTR is a SET and a set does not guarantee the order of its elements. According to the documentation, the toArray() function follows the iterator's order and the iterator returns elements in no particular order (unless this set is an instance of some class that provides a guarantee). This makes the test outcome non-deterministic. To fix this, the actual array needs to be sorted so that it can guarantee the order of its elements.Similar fix is made for the line
ConfigInitializerTest.testInitialize:71to remove the flakiness.Fix
Sorting the output returned by the toArray() function. This removes the non-determinism without needing any further changes and ensures that the flakiness of the test is removed.
CHANGESlog.