feat: log all matched mocks per failed testcase

[AkashKumar7902]

Related Issue

• Info about Issue or bug

Closes: #1562

Describe the changes you've made

if debug flag is enabled then all the mocks which are matched are logged in case of failed testcase.
Added the logic to remove unused mocks when removeUnusedMocks flag is enabled.

How I tested this feature ? (guide)

To test this feature I recorded two test-sets, one having 2 test-cases and other having 4 test-cases. 

I added body.ts as a noisy field in test-set-1 which will let it to pass and test-set-0 will fail. purpose of doing this is to demonstrate that --removeUnusedMocks only works when test-set passes.

Now lets remove a test-case, say test-4.yaml.

currently, in mocks.yaml of test-set-1, there are a total of 7 mocks

lets run keploy test command with --removeUnusedMocks flag set and see the log:

       ▓██▓▄
    ▓▓▓▓██▓█▓▄
     ████████▓▒
          ▀▓▓███▄      ▄▄   ▄               ▌
         ▄▌▌▓▓████▄    ██ ▓█▀  ▄▌▀▄  ▓▓▌▄   ▓█  ▄▌▓▓▌▄ ▌▌   ▓
       ▓█████████▌▓▓   ██▓█▄  ▓█▄▓▓ ▐█▌  ██ ▓█  █▌  ██  █▌ █▓
      ▓▓▓▓▀▀▀▀▓▓▓▓▓▓▌  ██  █▓  ▓▌▄▄ ▐█▓▄▓█▀ █▓█ ▀█▄▄█▀   █▓█
       ▓▌                           ▐█▌                   █▌
        ▓
  
version: 2-dev

🐰 Keploy: 2024-02-22T15:30:37+05:30    WARN    Delay is set to 5 seconds, incase your app takes more time to start use --delay to set custom delay
🐰 Keploy: 2024-02-22T15:30:37+05:30    INFO    Example usage: sudo -E env PATH=$PATH keploy test -c "/path/to/user/app" --delay 6
🐰 Keploy: 2024-02-22T15:30:37+05:30    INFO            {"keploy test and mock path": "/home/akash/Desktop/keploy/keploy"}
🐰 Keploy: 2024-02-22T15:30:37+05:30    INFO            {"keploy testReport path": "/home/akash/Desktop/keploy/keploy/testReports/test-run-4"}
🐰 Keploy: 2024-02-22T15:30:42+05:30    INFO    keploy initialized and probes added to the kernel.
🐰 Keploy: 2024-02-22T15:30:42+05:30    INFO    Java detected and CA already exists     {"path": "/usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts"}
🐰 Keploy: 2024-02-22T15:30:42+05:30    INFO    Java detected and CA already exists     {"path": "/usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts"}
🐰 Keploy: 2024-02-22T15:30:43+05:30    INFO    Keploy has hijacked the DNS resolution mechanism, your application may misbehave in keploy test mode if you have provided wrong domain name in your application code.
🐰 Keploy: 2024-02-22T15:30:43+05:30    INFO    Proxy started at port:16789
🐰 Keploy: 2024-02-22T15:30:43+05:30    INFO    starting DNS server at addr :16789
🐰 Keploy: 2024-02-22T15:30:43+05:30    INFO    running user application for    {"test-set": "test-set-0"}
PID: 175836
           [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

                                                                                                                      [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
                                                                - using env:    export GIN_MODE=release
                                                                                                        - using code:   gin.SetMode(gin.ReleaseMode)

       [GIN-debug] GET    /:param                   --> main.getURL (3 handlers)
                                                                                [GIN-debug] POST   /url                      --> main.putURL (3 handlers)
            [GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
                                                                                                             Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
                                                                    [GIN-debug] Listening and serving HTTP on :8080
                                                                                                                   🐰 Keploy: 2024-02-22T15:30:43+05:30      INFO            {"no of test cases": 2, "test-set": "test-set-0"}
🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO    starting test for of    {"test case": "test-1", "test set": "test-set-0"}
[GIN] 2024/02/22 - 15:30:48 | 200 |    2.255878ms |             ::1 | POST     "/url"
Testrun failed for testcase with id: "test-1"

--------------------------------------------------------------------

+-------------------------------------------------------------------------------------------------------------+
|                                                DIFFS TEST-1                                                 |
+-------------------------------------------------------------------------------------------------------------+
|                     EXPECT HEADER                    |                   ACTUAL HEADER                      |
| -----------------------------------------------------+----------------------------------------------------- |
|                                                      |                                                      |
|                                                                                                             |
|                      EXPECT BODY                     |                    ACTUAL BODY                       |
| -----------------------------------------------------+----------------------------------------------------- |
|    {                                                 |  {                                                   |
|   -  "ts": 1.7085915317954831e+18,                   | +  "ts": 1.7085960481977405e+18,                     |
|      "url": "http://localhost:8080/Lhr4BWAi"         |    "url": "http://localhost:8080/Lhr4BWAi"           |
|    }                                                 |  }                                                   |
|                                                      |                                                      |
|                                                                                                             |
+-------------------------------------------------------------------------------------------------------------+
🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO            {"matched mocks": ["mock-3"]}
🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO    result  {"testcase id": "test-1", "testset id": "test-set-0", "passed": "false"}
🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO    starting test for of    {"test case": "test-2", "test set": "test-set-0"}
[GIN] 2024/02/22 - 15:30:48 | 303 |     558.955µs |             ::1 | GET      "/Lhr4BWAi"
Testrun passed for testcase with id: "test-2"

--------------------------------------------------------------------

🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO    result  {"testcase id": "test-2", "testset id": "test-set-0", "passed": "true"}
🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO    test report for test-set-0:     {"name: ": "report-1", "path: ": "/home/akash/Desktop/keploy/keploy/report-1"}

 <=========================================> 
  TESTRUN SUMMARY. For testrun with id: "test-set-0"
        Total tests: 2
        Total test passed: 1
        Total test failed: 1
 <=========================================> 

🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO    keploy has initiated the shutdown of the user application.
🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO    keploy terminated user application
🐰 Keploy: 2024-02-22T15:30:48+05:30    INFO    running user application for    {"test-set": "test-set-1"}
PID: 175921
           [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

                                                                                                                      [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
                                                                - using env:    export GIN_MODE=release
                                                                                                        - using code:   gin.SetMode(gin.ReleaseMode)

       [GIN-debug] GET    /:param                   --> main.getURL (3 handlers)
                                                                                [GIN-debug] POST   /url                      --> main.putURL (3 handlers)
            [GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
                                                                                                             Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
                                                                    [GIN-debug] Listening and serving HTTP on :8080
                                                                                                                   🐰 Keploy: 2024-02-22T15:30:48+05:30      INFO            {"no of test cases": 3, "test-set": "test-set-1"}
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    starting test for of    {"test case": "test-1", "test set": "test-set-1"}
[GIN] 2024/02/22 - 15:30:53 | 200 |     1.27849ms |             ::1 | POST     "/url"
Testrun passed for testcase with id: "test-1"

--------------------------------------------------------------------

🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    result  {"testcase id": "test-1", "testset id": "test-set-1", "passed": "true"}
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    starting test for of    {"test case": "test-2", "test set": "test-set-1"}
[GIN] 2024/02/22 - 15:30:53 | 303 |     724.771µs |             ::1 | GET      "/Lhr4BWAi"
Testrun passed for testcase with id: "test-2"

--------------------------------------------------------------------

🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    result  {"testcase id": "test-2", "testset id": "test-set-1", "passed": "true"}
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    starting test for of    {"test case": "test-3", "test set": "test-set-1"}
[GIN] 2024/02/22 - 15:30:53 | 200 |     455.969µs |             ::1 | POST     "/url"
Testrun passed for testcase with id: "test-3"

--------------------------------------------------------------------

🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    result  {"testcase id": "test-3", "testset id": "test-set-1", "passed": "true"}
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    removed unused mocks from mock file     {"test-set": "test-set-1"}
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    test report for test-set-1:     {"name: ": "report-2", "path: ": "/home/akash/Desktop/keploy/keploy/report-2"}

 <=========================================> 
  TESTRUN SUMMARY. For testrun with id: "test-set-1"
        Total tests: 3
        Total test passed: 3
        Total test failed: 0
 <=========================================> 

🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    keploy has initiated the shutdown of the user application.

 <=========================================> 
  COMPLETE TESTRUN SUMMARY. 
        Total tests: 5
        Total test passed: 4
        Total test failed: 1

        Test Suite Name         Total Test      Passed          Failed

        "test-set-0"            2               1               1
        "test-set-1"            3               3               0
<=========================================> 

🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    test run completed      {"passed overall": false}
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    Exiting keploy program gracefully.
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    keploy terminated user application
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    eBPF resources released successfully...
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    Dns server stopped
🐰 Keploy: 2024-02-22T15:30:53+05:30    INFO    proxy stopped...

Looking at the logs you can notice that unused mocks are removed only after test-set-1, and not after failed test-set-0 as discussed above.
🐰 Keploy: 2024-02-22T15:30:53+05:30 INFO removed unused mocks from mock file {"test-set": "test-set-1"}

also now mocks.yaml has only 5 mocks instead of 6.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Code style update (formatting, local variables)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Please let us know if any test cases are added

Please describe the tests(if any). Provide instructions how its affecting the coverage.

Describe if there is any unusual behaviour of your code(Write NA if there isn't)

A clear and concise description of it.

Checklist:

• My code follows the style guidelines of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.

Screenshots (if any)

Original	Updated
original screenshot	updated screenshot

[charankamarapu]

Please address the comments and fill the PR description

[shivamsouravjha]

Did you check if any new race condition is introduced ?

[AkashKumar7902]

regarding the race condition, there were total 4 found data races, but none of them point to the files changed in this PR
https://pastebin.com/JncRn2k6

[shivamsouravjha]

LGTM!