MP Fault Tolerance 1.1 Feature Test Summary

[tevans78]

Existing functionality

Fault Tolerance (FT) 1.1 retains all the existing function of FT 1.0

• We will run all existing FT 1.0 tests against FT 1.1
• All the FT 1.0 tests (unit, FAT and TCK) are running automatically in the build
  • bucket com.ibm.ws.microprofile.faulttolerance.cdi_fat

Use of TCK

FT 1.1 will have a TCK which we have integrated into the build

• bucket com.ibm.ws.microprofile.faulttolerance.1.1_fat_tck

New features for Fault Tolerance 1.1

Add config options to disable each of the FT annotation types - 2707 - spec issue #109

• New TCK tests:
  • For each of @CircuitBreaker, @Retry, @Timeout, @Bulkhead, @Fallback, @Asynchronous:
    • ensure the annotation can be disabled at the method level
    • ensure the annotation can be disabled at the class level
    • ensure the annotation can be disabled globally
    • ensure the annotation can be disabled globally, and then enabled at the method level
    • ensure the annotation can be disabled globally, and then enabled at the class level
    • ensure the annotation can be disabled globally, and then enabled at the class level, and then disabled again at the method level
    • ensure the annotation can be disabled at the class level and then enabled at the method level
    • ensure that fault tolerance can be disabled globally but this can be overriden by enabling the annotation at the method level
    • ensure that fault tolerance annotations disabled with MP_Fault_Tolerance_NonFallback_Enabled=false can be enabled at the method level
    • ensure that fault tolerance annotations disabled with MP_Fault_Tolerance_NonFallback_Enabled=false can be enabled at the class level
    • ensure that fault tolerance annotations disabled with MP_Fault_Tolerance_NonFallback_Enabled=false can be enabled globally
  • Test that disabling a single annotation works correctly when more than one annotation is applied to the same method

Add getFailure() method to ExecutionContext - spec issue #224

• New TCK tests:
  • when a method fails and a fallback handler is called, ensure getFailure() returns the exception
  • the execution context is only available to the application in a fallback handler when a method has failed, so it doesn't make sense to test the value in a success case

Automated metrics integration - 2569 - spec issue #234

• Adding an FT annotation to a method should result in metrics automatically being gathered for that method
• There are quite a lot of metrics, so I'll describe the general approach for each metric type
• New TCK tests:
  • For each annotation type, check that the appropriate metrics are added when the annotation is applied to a method
  • For any one annotation type, check that the appropriate metrics are added to each method when the annotation is applied to a class
  • For all counter metrics, ensure they are incremented at the correct time
    • e.g. for a counter of failures, ensure it is incremented if a method throws an exception and not incremented if it returns successfully
  • For all gauges, get the system into a predictable state and check that the gauge reports the expected value
  • For histogram of execution times, run a series of executions with predictable execution times, check that the histogram quantiles are roughly as expected
  • For histogram of bulkhead queue wait times, fill the bulkhead with tasks so that executions can be added to the queue
  • Put all annotations on the same method and ensure that all metrics are added
    • as far as is practical, ensure that all metrics are updated correctly when this method is called
    • we're worried here about some interaction causing a metric to be updated twice, or somehow missed completely
  • Ensure metrics are not generated if metric integration is disabled

[Azquelt]

I've updated the first post with the feature test summary details. It is ready for review.

[sabolo]

@Azquelt : The feature test has 2 parts to it. The first describing the testing done and strategy which you have provided above. The second part is to provide a squad confidence level in the testing done. Please can you update the above with part 2. For your reference, the second part states

Part 2:
Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.

Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:
0 - No automated testing delivered
1 - We have minimal automated coverage of the feature including golden paths. There is a relatively high risk that defects or issues could be found in this feature.
2 - We have delivered a reasonable automated coverage of the golden paths of this feature but are aware of gaps and extra testing that could be done here. Error/outlying scenarios are not really covered. There are likely risks that issues may exist in the golden paths
3 - We have delivered all automated testing we believe is needed for the golden paths of this feature and minimal coverage of the error/outlying scenarios. There is a risk when the feature is used outside the golden paths however we are confident on the golden path. Note: This may still be a valid end state for a feature... things like Beta features may well suffice at this level.
4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.
5 - We have delivered all automated testing we believe is needed for this feature. The testing covers all golden path cases as well as all the error/outlying scenarios that make sense. We are not aware of any gaps in the testing at this time. No manual testing is required to verify this feature.

Based on your answer above, for any answer other than a 4 or 5 please provide details of what drove your answer. Please be aware, it may be perfectly reasonable in some scenarios to deliver with any value above. We may accept no automated testing is needed for some features, we may be happy with low levels of testing on samples for instance so please don't feel the need to drive to a 5. We need your honest assessment as a team and the reasoning for why you believe shipping at that level is valid. What are the gaps, what is the risk etc. Please also provide links to the follow on work that is needed to close the gaps (should you deem it needed)

Confidence:
Comments:

[sabolo]

@Azquelt : Please could I also ask that you update the test information above to add the test bucket names that you have added the tests to. It is unclear if the tests listed are new in the TCK or are delivered as a seperate bucket. Thanks

[Azquelt]

Updated to say that all new tests are added to the TCK, along with the FAT bucket name where the TCK is run in the liberty build.

Also discovered that the 1.0 FAT tests had not been updated to run against 1.1, I have #4638 going through now to make that change. I consider this to have a low risk of exposing problems because there's very little code exercised by the existing tests that isn't shared between 1.1 and 1.0. #4638 now merged

[Azquelt]

Test confidence score: 4
Comments: There are some edge cases around conflicting metric names being generated by different apps that we don't do much testing of. The only test we do have checks that this situation doesn't cause errors. The situation is fairly uncommon, you'd need to have two apps running on the same server which include the same class which has fault tolerance annotations on it.

[tevans78]

FT 1.1 is complete. Closing.