Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

intermittent failures in GP tests for 13.1.0 Release Candidate #2654

Closed
mdebarros opened this issue Jan 25, 2022 · 2 comments
Closed

intermittent failures in GP tests for 13.1.0 Release Candidate #2654

mdebarros opened this issue Jan 25, 2022 · 2 comments
Assignees
@mdebarros
Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it
Milestone
Sprint 17.1

Comments

@mdebarros
Copy link
Member

mdebarros commented Jan 25, 2022 •
edited by elnyry-sam-k
Loading

Summary:

13.1.0 Release Candidate tests are not stable with the introduction of CGS test into GP due to the async nature of the Centra-Settlement component.

Moja1 environment runs a daily GP test job. Check the #announcements channel in the Mojaloop slack org to see test result history showing the intermittent failures.

Severity:
Medium

Priority:
Medium

Expected Behavior

  • GP tests in 13.1.0 should be consistent and as deterministic as possible

Steps to Reproduce

  1. Install 13.1.0 release candidate
  2. Run helm test command several times

Specifications

  • Component (if known): Helm
  • Version: v13.1.0 Release candidate (13.1.0-13148.8b8fd41)
  • Platform: n/a
  • Subsystem: n/a
  • Type of testing: n/a
  • Bug found/raised by: @mdebarros

Notes:

Investigating 1/31:

  1. Timing issue on settlement tests (retry logic)
  2. Unknown issue, additional failures (inapplicable windows error)
  • Severity when opened: Medium
  • Priority when opened: Medium
@mdebarros mdebarros added bug Something isn't working or it has wrong behavior on a Mojaloop Core service oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it labels Jan 25, 2022
@elnyry-sam-k elnyry-sam-k changed the title Unstable GP tests for 13.1.0 Release Candidate 2 intermittent failures in GP tests for 13.1.0 Release Candidate Jan 25, 2022
@mdebarros mdebarros self-assigned this Jan 31, 2022
@mdebarros mdebarros changed the title 2 intermittent failures in GP tests for 13.1.0 Release Candidate intermittent failures in GP tests for 13.1.0 Release Candidate Jan 31, 2022
@elnyry-sam-k elnyry-sam-k added this to the Sprint 17.1 milestone Feb 3, 2022
@mdebarros mdebarros mentioned this issue Feb 7, 2022
Merged
@mdebarros
Copy link
Member Author

mdebarros commented Feb 8, 2022 •
edited
Loading

The TTK Test-Case v13.1.0 release has been re-released with the associated fix: https://github.com/mojaloop/testing-toolkit-test-cases/releases/tag/v13.1.0

The following consecutive tests have passed over the last 9 days, with zero failures:

Note: We previously saw around a 40% failure rate prior to adding retry logic into the test cases.

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP Tests
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Mon, 31 Jan 2022 13:40:07 GMT
Completed time: Mon, 31 Jan 2022 13:42:57 GMT
Runtime duration: 169576 ms

View Report

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP CronJob
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Tue, 01 Feb 2022 08:00:11 GMT
Completed time: Tue, 01 Feb 2022 08:02:54 GMT
Runtime duration: 162948 ms

View Report

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP CronJob
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Wed, 02 Feb 2022 08:00:07 GMT
Completed time: Wed, 02 Feb 2022 08:02:54 GMT
Runtime duration: 166548 ms

View Report

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP CronJob
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Thu, 03 Feb 2022 08:00:04 GMT
Completed time: Thu, 03 Feb 2022 08:02:38 GMT
Runtime duration: 153854 ms

View Report

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP CronJob
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Fri, 04 Feb 2022 08:00:22 GMT
Completed time: Fri, 04 Feb 2022 08:03:09 GMT
Runtime duration: 167730 ms

View Report

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP CronJob
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Sat, 05 Feb 2022 08:00:09 GMT
Completed time: Sat, 05 Feb 2022 08:02:53 GMT
Runtime duration: 164083 ms

View Report

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP CronJob
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Sun, 06 Feb 2022 08:00:04 GMT
Completed time: Sun, 06 Feb 2022 08:02:39 GMT
Runtime duration: 154979 ms

View Report

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP CronJob
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Mon, 07 Feb 2022 08:00:19 GMT
Completed time: Mon, 07 Feb 2022 08:02:52 GMT
Runtime duration: 153140 ms

View Report

Testing Toolkit Report
Test Result: PASSED
Test Suite: GP CronJob
Environment: moja1.test.mojaloop.live
Total assertions: 2318
Passed assertions: 2318
Failed assertions: 0
Total requests: 551
Total test cases: 120
Passed percentage: 100.00%
Started time: Tue, 08 Feb 2022 08:00:12 GMT
Completed time: Tue, 08 Feb 2022 08:02:52 GMT
Runtime duration: 160539 ms

View Report

mdebarros added a commit to mojaloop/testing-toolkit-test-cases that referenced this issue Feb 8, 2022
@mdebarros
fix(mojaloop/#2654): intermittent failures in gp tests for 13.1.0 rel…
a4312e8 
…ease candidate (#60)

* fix(mojaloop/#2654): intermittent failures in gp tests for 13.1.0 release candidate

Fix for [mojaloop/project/issues/2654](mojaloop/project#2654):
- added retry logic to several test-cases within the CGS settlements test scenarios to improve consistency
- added the following environment variables which are only used by the following test collection: [collections/hub/golden_path/settlement_cgs/newsetcgs.json](https://github.com/mojaloop/testing-toolkit-test-cases/pull/60/files#diff-0d6c451f8c70def565cbd0e4460fb06d4f83ac6cb73cb83f0cb45318503f6b14)
    - RETRY_MAX_ATTEMPTS (default: 10): This is the number of attempts that should be retried, this is currently used when validating the settlement/position accounts after the settlement is completed in the above test collection
    - RETRY_MAX_WAIT_IN_MS (default: 200): This is the wait time before retry attempts in milliseconds.
@elnyry-sam-k
Copy link
Member

Closing this after a week of passing results; most recent one: https://mojaloop.slack.com/archives/CG3MAJZ5J/p1644316664519559

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdebarros mdebarros

Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it
Projects
None yet
Milestone
Sprint 17.1
Development

No branches or pull requests
2 participants
@elnyry-sam-k @mdebarros