feat: add support for keploy test bench for native environment

[gouravkrosx]

Related Issue

• To Test keploy's codebase (majorly integrations), keploy should have its testing framework.

Closes: #1739, #1740, #1792, #1793, #1801
Linked PR: keploy/pilot#1, https://github.com/keploy/ebpf/pull/41

Describe the changes you've made

• Using the --enableTesting flag, you can turn on the testing env of keploy.

Approach

• Keploy's test bench is built on the principle that Keploy can test itself, operating in two distinct modes: RECORD and TEST. Each mode activates different sections of the codebase, with some overlap between the two. These modes can be executed concurrently to enhance testing efficiency.

• keployR & keployT are built from the same binary just the name is different for readability.

• For example, if you have recorded test cases and mocks for a Go + MongoDB application, the test bench allows you to run Keploy in test mode (keployT) from within Keploy in record mode (keployR) using the following pseudo command: keployR record -c "keployT test -c ./gin-mongo-app". This command initiates Keploy in record mode, which in turn launches Keploy in test mode to run the application. As keployT executes the test sets, it simulates API calls for each test case. These simulated API calls are then recorded as test cases by keployR, along with any external calls from the application being captured as mocks. Meanwhile, keployT generates a final report of the test runs. This approach allows for the simultaneous execution of both Keploy's record and test flows.

• One significant benefit of this method is that it eliminates the need to set up external dependencies in the CI pipeline for testing. KeployT acts as a virtual database, with its calls being recorded by keployR, streamlining the testing process.

Caveats:

• Port Configuration: The ports for Keploy Record (keployR) and Keploy Test (keployT) need to be hardcoded.
• Sequential Test Sets: Only one test set can be run at a time.
• Limited Environment Support: This feature is currently available only for native binary environments and not for Docker environments.
• Recording Delay: After the test runs, there is a waiting period of 2 seconds to ensure proper recording of test cases.
• Process Filtering: Internally, Keploy Record (keployR) should handle only application-related calls and not Keploy Test (keployT) related calls. To achieve this, keployT waits for keployR to retrieve the PID before starting the test run. This ensures that keployR can filter out keployT-related calls based on the PID.
• As of now, running this setup on WSL is not handled.

Steps to run:

• Get this branch locally and build keploy using go build -tags=viper_bind_struct -o keploy . && sudo mv keploy /usr/local/bin/keploy

• Take any application, I've tested for the gin-mongo sample app so you can take that one as of now. And record test some cases, make at least two sessions of recording. Use cmd: sudo -E env PATH=$PATH keploy record -c "./ginApp"

  Recording of test cases via test-bench

  • Make two copies of the same binary for easy readability of the command using sudo cp /usr/local/bin/keploy /usr/local/bin/keployr && sudo cp /usr/local/bin/keploy /usr/local/bin/keployt
  • Run this command to record test cases (for each test-set you have to run this command) : sudo -E env PATH=$PATH keployr record -c "sudo -E env PATH=$PATH keployt test -c ./ginApp --proxyPort 56789 --dnsPort 46789 --delay=10 --testsets test-set-0 --enableTesting" --path "./test-bench/" --proxyPort=36789 --dnsPort 26789 --enableTesting
  • This command will generate new test cases from your existing pre-recorded test cases.

  Tests and Mocks Assertion

  • Before Asserting test cases, you can run the newly generated test cases in test mode using: sudo -E env PATH=$PATH keploy test -c "./ginApp" --path "./test-bench/"
  • Now for tests and mocks assertion, you will be requiring another helper repo.
  • Build binary of this testing pilot using, go build -o pilot . && sudo mv pilot /usr/local/bin
  • Assert Tests using: pilot -test-assert -preRecPath ./ -testBenchPath ./test-bench ( you can also provide a config path to consider noisy parameters using -configPath ./)
  • Prepare Mock to assert: pilot -mock-assert -preRecPath ./ -testBenchPath ./test-bench
  • Do the actual mock assertion: for this, you just need to run test mode for both the pre-recorded and newly recorded tests and mocks"
  • If the behavior of this approach is similar to the normal recording and testing of keploy. Then you are good to go.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Code style update (formatting, local variables)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Please let us know if any test cases are added

Please describe the tests(if any). Provide instructions how its affecting the coverage.

Describe if there is any unusual behaviour of your code(Write NA if there isn't)

NA

Checklist:

• My code follows the style guidelines of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.

Screenshots (if any)

Original	Updated
original screenshot	updated screenshot

[charankamarapu]

Please address comments and provide steps on how developers can run this on local machines and fix the pipeline

[charankamarapu]

LGTM

[slayerjain]

@gouravkrosx lets just call this pilot - https://github.com/keploy/Ktest-Pilot

[slayerjain]

I tried this, and its awesome!