ReṼoman (Rev-Woman)

---

ReṼoman is an API automation tool for JVM (Java/Kotlin) from the API-first SaaS company, Salesforce. It re-imagines API automation by letting you execute a Postman collection in a JVM program/test.

---

To start with, think of it as Postman for JVM (Java/Kotlin); that emulates the Postman Collection Runner through a Java program, essentially translating your manual testing into Automation, without any loss or resistance. But it’s even better!

It strikes a balance between flexibility provided by low-level tools like REST Assured and ease of use provided by UI tools like Postman

Artifact

Maven

<dependency>
  <groupId>com.salesforce.revoman</groupId>
  <artifactId>revoman</artifactId>
  <version>0.41.6</version>
</dependency>

Bazel

"com.salesforce.revoman:revoman"

Gradle Kts

implementation("com.salesforce.revoman:revoman:0.41.6")

Table of Contents

• Artifact
• Why ReṼoman?
  • The Problem
  • The Solution
• API automation with ReṼoman
  • Template-Driven Testing
  • A Simple Example
  • Rundown
  • Advanced Example
• Debugging UX
• Features
  • Type Safety with flexible JSON ← → POJO marshalling/serialization and unmarshalling/deserialization
  • Execution Control
  • Pre- and Post-Hooks
  • Pre-req and Tests scripts
  • Mutable Environment
  • Compose Modular Executions
  • Custom Dynamic variables
• USP
  • Low-code
  • CI/CD integrability
  • Up-to-date Postman collections that live along with Code in VCS
  • Unified framework for Automating Persona-based Manual testing
• Perf
• Future
• FAQs
  • How to Debug a step in the middle of an Execution?
  • Is there a way to Mark a Postman collection Step?
  • Do I need to migrate all my existing TestUtils to Postman Collections?
  • Why not use Newman or Postman CLI?
• 🙌🏼 Consume-Collaborate-Contribute

Tech-Talk given at Open Source Conf - 2023, 🎴Slide deck

Why ReṼoman?

The Problem

• The majority of JVM SaaS applications are REST-based. But the API automation is done through a Mul-T-verse of Integration/Functional tests, E2E tests and Manual tests, each with its own frameworks, tools, and internal utilities, testing almost the same code flow.

• These custom alien automation frameworks, often built using low-level tools like REST Assured, are specific to a service or domain and are rigid to reuse, extend and difficult to maintain.

• This automation competes on cognitive complexity and learning curve with the Prod code, and mostly, automation wins.

• After a point, the API automation may deviate from its purpose of augmenting real end-user interaction and turns into a foot-chain for development.

The Solution

Contrary to these custom frameworks, almost every team uses Postman for manual testing their APIs. Postman collections contain a lot of information about your APIs and the order in which they need to be executed for manual testing, in a Structured Template. Leveraging it can mitigate writing a lot of code as we translate those manual steps into automation.

• How productive would it be, if you can plug your exported Postman collection template, that you anyway would have created for your manual testing, and execute them through your JVM tests?

• How about a Universal API automation tool that promotes low code and low-cognitive-complexity and strikes a balance between flexibility and ease of use?

API automation with ReṼoman

Template-Driven Testing

• The exported Postman collection JSON file is referred to as a Postman template, as it contains some placeholders/variables in the {{variable-key}} pattern. You can read more about it here

• ReṼoman understands these templates and replaces these variables at the runtime, similar to Postman. It supports

  • Nested variables, e.g., {{variable-key{{nested-variable-key}}}}

  • Dynamic variables, e.g., {{$randomUUID}}, {{$randomEmail}}

  • Custom Dynamic variables

You can kick off this Template-Driven Testing by invoking ReVoman.revUp(), supplying your Postman templates and environments, and all your customizations through a configuration:

final var rundown =
  ReVoman.revUp(
    Kick.configure()
     ...
    .off())

A Simple Example

Here is a simple Exported Postman collection and Environment, to hit a free public RESTFUL-API. You can import and manually test this collection through the Run collection button like this:

You can automate the same using ReṼoman in a Junit test by supplying the template and environment path:

RestfulAPIDevTest.java

@Test
@DisplayName("restful-api.dev")
void restfulApiDev() {
  final var rundown =
    ReVoman.revUp( // (1)
      Kick.configure()
          .templatePath(PM_COLLECTION_PATH) // (2)
          .environmentPath(PM_ENVIRONMENT_PATH) // (3)
          .off());
  assertThat(rundown.stepReports).hasSize(3); // (4)
}

1. revUp is the method to call passing a configuration, built as below

2. Supply an exported Postman collection JSON file path

3. Supply an exported Postman environment JSON file path

4. Run more assertions on the Rundown

Rundown

After all this, you receive back a detailed Rundown in return. It contains everything you need to know about what happened in an execution, such that you can seamlessly run more assertions on top of the run.

Rundown(
  stepNameToReport: List<StepReport>,
  environment: PostmanEnvironment)

StepReport(
  step: Step,
  requestInfo: Either<RequestFailure, TxnInfo<Request>>? = null, // (1)
  preHookFailure: PreHookFailure? = null,
  responseInfo: Either<ResponseFailure, TxnInfo<Response>>? = null,
  postHookFailure: PostHookFailure? = null,
  envSnapshot: PostmanEnvironment<Any?> // (2)
)

1. Either type from the VAVR library represents either of the two states, error or success

2. Snapshot of Environment at the end of each step execution. It can be compared with previous or next step environment snapshots to see what changed in this step

Rundown has many convenience methods to ease applying further assertions on top of it.

💡	Other simple examples to see in Action: PokemonTest.java

Advanced Example

ReṼoman isn’t just limited to executing Collection like Postman; you can add more bells & whistles 🔔:

PQE2EWithSMTest.java

final var pqRundown =
  ReVoman.revUp( // (1)
      Kick.configure()
          .templatePaths(PQ_TEMPLATE_PATHS) // (2)
          .environmentPath(PQ_ENV_PATH) // (3)
          .dynamicEnvironment( // (4)
              Map.of(
                  "$quoteFieldsToQuery", "LineItemCount, CalculationStatus",
                  "$qliFieldsToQuery", "Id, Product2Id",
                  "$qlrFieldsToQuery", "Id, QuoteId, MainQuoteLineId, AssociatedQuoteLineId"))
          .customDynamicVariableGenerator( // (5)
              "$unitPrice",
              (ignore1, ignore2, ignore3) -> String.valueOf(Random.Default.nextInt(999) + 1))
          .nodeModulesRelativePath("js") // (6)
          .haltOnFailureOfTypeExcept(
              HTTP_STATUS,
              afterAllStepsContainingHeader("ignoreHTTPStatusUnsuccessful")) // (7)
          .requestConfig( // (8)
              unmarshallRequest(
                  beforeAllStepsWithURIPathEndingWith(PQ_URI_PATH),
                  PlaceQuoteInputRepresentation.class,
                  adapter(PlaceQuoteInputRepresentation.class)))
          .responseConfig( // (9)
              unmarshallResponse(
                  afterAllStepsWithURIPathEndingWith(PQ_URI_PATH),
                  PlaceQuoteOutputRepresentation.class),
              unmarshallResponse(
                  afterAllStepsWithURIPathEndingWith(COMPOSITE_GRAPH_URI_PATH),
                  CompositeGraphResponse.class,
                  CompositeGraphResponse.ADAPTER))
          .hooks( // (10)
              pre(
                  beforeAllStepsWithURIPathEndingWith(PQ_URI_PATH),
                  (step, requestInfo, rundown) -> {
                    if (requestInfo.containsHeader(IS_SYNC_HEADER)) {
                      LOGGER.info("This is a Sync step: {}", step);
                    }
                  }),
              post(
                  afterAllStepsWithURIPathEndingWith(PQ_URI_PATH),
                  (stepReport, ignore) -> {
                    validatePQResponse(stepReport); // (11)
                    final var isSyncStep =
                        stepReport.responseInfo.get().containsHeader(IS_SYNC_HEADER);
                    if (!isSyncStep) {
                      LOGGER.info(
                          "Waiting in PostHook of the Async Step: {}, for the Quote's Asynchronous processing to finish",
                          stepReport.step);
                      // ! CAUTION 10/09/23 gopala.akshintala: This can be flaky until
                      // polling is implemented
                      Thread.sleep(5000);
                    }
                  }),
              post(
                  afterAllStepsWithURIPathEndingWith(COMPOSITE_GRAPH_URI_PATH),
                  (stepReport, ignore) -> validateCompositeGraphResponse(stepReport)),
              post(
                  afterStepName("query-quote-and-related-records"),
                  (ignore, rundown) -> assertAfterPQCreate(rundown.mutableEnv)))
          .globalCustomTypeAdapter(new IDAdapter()) // (12)
          .insecureHttp(true) // (13)
          .off()); // Kick-off
assertThat(pqRundown.firstUnIgnoredUnsuccessfulStepReport()).isNull(); // (14)
assertThat(pqRundown.mutableEnv)
  .containsAtLeastEntriesIn(
      Map.of(
          "quoteCalculationStatusForSkipPricing", PricingPref.Skip.completeStatus,
          "quoteCalculationStatus", PricingPref.System.completeStatus,
          "quoteCalculationStatusAfterAllUpdates", PricingPref.System.completeStatus));

1. revUp() is the method to call passing a configuration, built as below

2. Supply the path (relative to resources) to the Template Collection JSON file

3. Supply the path (relative to resources) to the Environment JSON file

4. Supply any dynamic environment that is runtime-specific

5. Custom Dynamic variables

6. Pre-req and Tests scripts

7. Execution Control

8. Request Config

9. Response Config

10. Pre- and Post-Hooks

11. Response Validations

12. Global Custom Type Adapters

13. Ignore Java cert issues when firing HTTP calls

14. Run more assertions on the Rundown

Debugging UX

This tool has particular emphasis on Debugging experience. Here is what a debugger view of a Rundown looks like:

🔍 Let’s zoom into a detailed view of one of those Step reports, which contains complete Request and Response info along failure information if any:

Here are the environment key-value pairs accumulated along the entire execution and appended to the environment from file and dynamicEnvironment supplied:

If something goes wrong at any stage during the Step execution, ReṼoman fails-fast and captures the Failure in StepReport:

Here is the failure hierarchy of what can go wrong in this process

ReṼoman logs all the key operations that happen inside its source-code. Watch your console to check what’s going on in the execution or troubleshoot from CI/CD logs

ℹ️	📝Sample log printed during execution

Features

Type Safety with flexible JSON ← → POJO marshalling/serialization and unmarshalling/deserialization

• ReṼoman internally uses a modern JSON library called Moshi

• There may be a POJO that inherits or contains legacy types which are hard or impossible to serialize. ReṼoman lets you serialize such types through globalSkipTypes, where you can filter-out these legacy types from Marshalling/Unmarshalling, only focussing on fields that matter.

• The payload may not map to POJO, and you may need a custom types adapter for Marshalling/Unmarshalling. Moshi has it covered for you with its advanced adapter mechanism and ReṼoman accepts Moshi adapters via

  • requestConfig — For types present as part of request payload for qualified Steps

  • responseConfig — For types present as part of response payload for qualified Steps

  • globalCustomTypeAdapters — For types present as part of request payload anywhere

• ReṼoman also comes bundled with JSON Reader utils and JSON Writer utils to help build Moshi adapters.

💡	Refer ConnectInputRepWithGraphAdapter for an advanced adapter use-case

JSON POJO Utils

JSON POJO Utils can be used to directly convert JSON to POJO and vice versa.

💡	See in Action: JsonPojoUtilsTest JsonPojoUtils2Test

Execution Control

The configuration offers methods through which the execution strategy can be controlled without making any changes to the template:

• haltOnAnyFailure — This defaults to false. If set to true, the execution fails-fast when it encounters a failure.

• haltOnFailureOfTypeExcept — This accepts pairs of ExeType and a PostTxnStepPick which are used to check if a Step can be ignored for failure for a specific failure type

• runOnlySteps, skipSteps — All these accept a predicate of type ExeStepPick, which is invoked passing the current Step instance to decide whether to execute or skip a step.

  • There are some ExeStepPick predicates bundled with ReṼoman under ExeStepPick.PickUtils e.g withName, inFolder etc. You can write a custom predicate of your own too.

Pre- and Post-Hooks

A hook lets you fiddle with the execution by plugging in your code before or after a Step execution.

You can pass a PreTxnStepPick/PostTxnStepPick which is a Predicate used to qualify a step for Pre-/Post-Hook respectively. ReṼoman comes bundled with some predicates under the namespace PreTxnStepPick.PickUtils/PostTxnStepPick.PickUtils e.g beforeAllStepsWithURIPathEndingWith, afterStepName etc. If those don’t fit your needs, you can write your own custom predicates too.

.hooks(
  pre(
      PreTxnStepPick,
      (currentStepName, requestInfo, rundown) -> {
        //...callback-code...
      }),
  post(
      PostTxnStepPick,
      (currentStepName, rundown) -> {
        //...callback-code...
      })
)

You can do things like assertion on the rundown, response validation, or environment and even mutate the environment, such that the execution of subsequent steps picks up those changes.

Plug-in your Java code in-between Postman execution

You can plug in your java code to create/generate values for environment variables which can be populated and picked-up by subsequent steps. For example, you may want some xyzId but you don’t have a Postman collection to create it. But instead, you have a Java utility to generate/create it. You can invoke the utility in a pre-hook of a step and set the value in rundown.mutableEnv, so the later steps can pick up value for {{xyzId}} variable from the environment.

Response Validations

• Post-Hooks are the best place to validate response right after the step.

• If you have configured a strong type for your response through responseConfig, you can write type-safe validations by extracting your Strong type Object using stepReport.responseInfo.get().<TypeT>getTypedTxnObj() (if you have configured responseConfig() or globalCustomTypeAdapters()) or use JsonPojoUtils.jsonToPojo(TypeT, stepReport.responseInfo.get().httpMsg.bodyString()) to convert it inline.

• If your response data structure is non-trivial and have requirements to execute validations with different strategies like fail-fast or error-accumulation, consider using a library like Vador

Pre-req and Tests scripts

• Postman lets you write custom javascript in Pre-req and Tests tabs that get executed before and after a step respectively. When you export the collection as a template, these scripts also come bundled.

• ReṼoman can execute this javascript on JVM. This support ensures that the Postman collection used for manual testing can be used as-is for the automation also, without any resistance to modify or overhead of maintaining separate versions for manual and automation.

  • Pre-req JS is executed as the first step before Unmarshall request.

  • Tests JS is executed right after receiving an HTTP response.

• ReṼoman supports using npm modules inside your Pre-req and Tests javascript. You can install npm modules in any folder and supply in the Kick config, the relative path of the parent folder that contains the node_modules folder using nodeModulesRelativePath(…​). Use those npm modules inside your scripts with require(…​), for example:

var moment = require("moment");
pm.environment.set("$currentDate", moment().format(("YYYY-MM-DD")));
var futureDateTime = moment().add(365, 'days');
pm.environment.set('$randomFutureDate', futureDateTime.format('YYYY-MM-DD'));

pm.environment.set("$quantity", _.random(1, 10)); // lodash doesn't need `require`

💡	node_modules Adds a lot of files to check in. You may replace them with a single distribution file If node_modules is ignored on your git repo, you can force-add to check in using the command git add -f <path>/node_modules

🔥	The recommendation is not to add too much code in Pre-req and Tests scripts, as it is not intuitive to troubleshoot. Use it for simple operations like set environment variables and use Post-Hooks JVM code for any non-trivial operations.

Mutable Environment

• Environment is the only mutable-shared state across step executions, which can be used for data-passing between consumer and the library.

• This can be mutated (set key-value pairs) through Pre-req and Tests scripts (using pm.environment.set()) and Pre-/Post-Hooks (using the reference rundown.mutableEnv) during execution.

pmEnvSnapshot in each StepReport

Each StepReport also has a pmEnvSnapshot to assert if a step has executed as expected and compare snapshots from different steps to examine the execution progress.

Compose Modular Executions

• You don’t have to squash all your steps into one mega collection. Instead, you can break them into easy-to-manage modular collections. ReVoman.revUp() accepts a list of collection paths through templatePaths()

• But that doesn’t mean you have to execute all these templates in one-go. You can make multiple ReVoman.revUp() calls for different collection.

• If you wish to compose these executions, you can do so by adding the previous execution’s mutableEnv to current execution using dynamicEnvironment parameter. This also comes in handy when you wish to execute a common step (e.g. UserSetup) inside a test setup method and use that environment for all the tests.

Custom Dynamic variables

If the in-built dynamic variables don’t fit your need, you can plug your own dynamic variable generator to be invoked to generate a value for your custom variable-key in the template at runtime.

USP

Low-code

💡	Here is an example of a low-code E2E test that automates ~75 steps

Compared to a traditional Integration/Functional or E2E test, approximately, the amount of code needed is 89% less using ReṼoman. The above test doesn’t just have low code, but also low in Cognitive Complexity and Transparent in what it does.

CI/CD integrability

• ReṼoman is like any JVM library that you can plug into any JVM program/test (e.g., JUnit tests or Integration tests).

• Apart from adding a dependency in the build tool, there is no extra setup needed to execute these tests with ReṼoman in CI/CD.

Up-to-date Postman collections that live along with Code in VCS

• A nice side effect is, this lets the Postman collections always stay up to date and the entire Postman collection guards each check-in in the form of a Test suite augmenting manual testing.

• Any day, you can find an up-to-date Postman collection for every feature you need to test, right in your VCS (Git) along with your code. Developers can import these templates directly from VCS into Postman for manual testing. This comes in very handy during a team blitz.

Unified framework for Automating Persona-based Manual testing

• ReṼoman brings a Unified & Simplified Test strategy across the mul-T-verse (Integration Tests, E2E Tests, and Manual testing with Postman) for any API.

• The automation stays as close as possible to Persona-based Manual testing, leading to Transparency and better Traceability of issues

• This forces engineers to think like API-first customers while writing tests.

• Test Data setup: You can use the ReṼoman for the Test data setup too. This eliminates the need for different teams to write their own internal utilities for data setup.

• E2E Test can even reside outside the Service repo, as long as it can hit the service API

Perf

This entire execution of ~75 steps, which includes 10 async steps, took a mere 122 sec on localhost. This can be much better on auto-build environments.

⚠️	ReṼoman internally is very light-weight, and the execution times are proportional to how your server responds or your network speed.

Future

The future looks bright with multiple impactful features in the pipeline:

• API metrics and Analytics

• It’s built with extensibility in mind. It can easily be extended to support other template formats, e.g., Kaiju templates used for availability testing.

• In-built polling support for Async steps

• Payload generation

• Flow control through YAML config

FAQs

How to Debug a step in the middle of an Execution?

• You can add a pre-hook to the Step you are interested and add a debug point inside that. This gets hit before ReṼoman fires the request in that Step

• You can get more adventurous by attaching revoman jar sources and directly adding conditional debug points inside this library source-code. You can search for logs in the source-code that indicate key operations to add conditional debug points with conditions like StepName etc.

Is there a way to Mark a Postman collection Step?

• You can add key-value pairs to a Step Header (e.g., ignoreHTTPStatusUnsuccessful=true).

• You can use this information in Step Picks or Pre- and Post-Hooks to identify a particular step to execute any conditional logic

Do I need to migrate all my existing TestUtils to Postman Collections?

You don’t have to. This is a JVM first tool, and you can interlace your TestUtils through Pre-/Post-Hooks

Why not use Newman or Postman CLI?

• ReṼoman may be similar to Newman or Postman CLI when it comes to executing a Postman collection, but the similarities end there.

• Newman or Postman CLI are built for node and cannot be executed within a JVM. Even if you are able to run with some hacky way, there is no easy way to assert results.

• ReṼoman is JVM first that lets you configure a lot more, and gives you back a detailed report to assert in a typesafe way

🙌🏼 Consume-Collaborate-Contribute

• This CONTRIBUTING doc has all the information to set up this library on your local and get hands-on.

• Any issues or PRs are welcome! ♥️

• Join this Slack Community to discuss issues or PRs related to Consumption-Collaboration-Contribution