MP Reactive Messaging 3.0 Feature Test Summary

[abutch3r]

Test Strategy

Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature.

For any feature, be aware that only FAT tests (not unit or BVT) are executed in our cross platform testing. To ensure cross platform testing ensure you have sufficient FAT coverage to verify the feature.

If delivering tests outside of the standard Liberty FAT framework, do the tests push the results into cognitive testing database (if not, consult with the CSI Team who can provide advice and verify if results are being received)?

List of FAT projects affected

Existing

• com.ibm.ws.microprofile.reactive.messaging_fat
• com.ibm.ws.microprofile.reactive.streams.operators_fat

New

• io.openliberty.microprofile.reactive.messaging.internal_fat
• io.openliberty.microprofile.reactive.messaging30.internal_fat_tck
• io.openliberty.microprofile.reactive.streams.operators30.internal_fat_tck

Test strategy

• What functionality is new or modified by this feature?

  • Reactive Messaging
    • Emitters support added 70
      • TCKs cover the use of Emitters, both positive and negative scenarios.
        • Includes testing of back pressure(@OnOverflow) strategies
      • Add FATs for Liberty-kafka handling requests from emitters
        • Cover Message and payload scenarios
        • REST Handling of Completion stages from emitted payloads.
    • Nack support for negative acknowledgement 98
      • TCKs covers payload, message and emitter based scenerios
      • Add FATs to cover Liberty-kafka handling of nacks
      • Add FATs to cover emitter REST scenarios where nacks occur.
    • Assembly validation on app start 119
      • TCKs validate that the correct exception(DeploymentException) is thrown for all invalid configurations.
      • Add FATs for validating associated messages that cause the deployment exceptions given these are implementation specific a
    • mpMetrics Integration 31
      • TCKs cover basic integration between mpReactive Messaging and mpMetrics ensuring counts are correct
      • Add FATs for more extensive testing of integration of metrics from external requests
    • Messaging and OpenTracing or OTEL
      • Add FATs for testing interactions between OpenTracing/OTEL with Reactive Messaging and Stream Operators when they are enabled
    • InstantOn support 26966
      • Add a FAT test that uses a Kafka server for messaging. It has Incoming and outgoing channel definitions connected to kafka. These are checkpointed with valid config keys, but invalid config values and when restored, it is provided with the valid config and sends and receives messages from the connected kafka server.
  • RSO does not have any new functionality
  • liberty-kafka connector
    • fast.ack 26692
      • Update FATs that are affected by the issue to cover and improve ability to capture this event.
        • RM 1.0 maintains existing behavior of the connector, while 3.0 uses the new behavior and as the connector code is common between the two we do cover both scenarios.
    • context.service 26678
      • FATs will be added that cover the various server related concurrency settings (default vs. conncurent-x.y), covering default, concurrent and invalid values

• Repeats

  • Add repeats to the existing FAT buckets to run against RM and RSO 3.0 for related MP 5.0 and 6.1 features
  • New buckets will run only RM and RSO 3.0 against MP 5.0 and 6.1
  • A select number of existing and new RM and RSO tests will be run against MP 6.0 as well as 1.0, 5.0 and 6.1 as the only component that does have an effect is MPConfig, which is a minor update. But, not repeating every test for MP 6.0 will help reduce the time it takes for a FULL fat run to complete.
  • New TCKs will be repeated for MP 5.0, 6.0 and 6.1

• What manual tests are there (if any)? (Note: Automated testing is expected for all features with manual testing considered an exception to the rule.)

  • No manual testing

Confidence Level

Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.

Please indicate your confidence in the testing (up to and including FAT) delivered with this feature by selecting one of these values:

4 - We have delivered all automated testing we believe is needed for the golden paths of this feature and have good coverage of the error/outlying scenarios. While more testing of the error/outlying scenarios could be added we believe there is minimal risk here and the cost of providing these is considered higher than the benefit they would provide.

[ayoho]

LGTM 👍