New Feature request - Generate tests from OpenAPI / Swagger yaml or json file

[9fingerdev]

I would like to see a new feature that creates basic tests for api calls based on OpenAPI/Swagger definition.

[sdnts]

@9fingerdev Could you elaborate a little on what you mean by basic tests?

[9fingerdev]

Thanks for your reply. One example is a rest api that sends an email and requires the following data: - Oauth2 token - email from address - email to address - subject - message I'm looking for a solution that would create tests with expected results for the happy path along with test that omit some off the data. Expecting the solution to create the test from OpenAPI/Swagger definition file. Is that something that APIMATIC can do?

…

-Troy

On Fri, Dec 1, 2017 at 5:24 AM, Siddhant Sinha ***@***.***> wrote: @9fingerdev <https://github.com/9fingerdev> Could you elaborate a little on what you mean by basic tests? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/postmanlabs/postman-app-support/issues/3877#issuecomment-348481862>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AghysMzbG2q27nc-IAZe6vlzTZcM-2Kpks5s7-_zgaJpZM4Qvkgn> .

[daraul]

I'd love to be able to define my postman/newman tests in my openapi spec, then have them brought into my collection after conversion.

[pauloallex24]

I'd love to be able to define my postman/newman tests in my openapi spec, then have them brought into my collection after conversion.

+1

[thim81]

Idea: Would a good approach be to generate "basic" tests, that can be defined in a config file?
Another option could be to have a config option and this this will generate basic tests, which would just work or that can be refined by the postman users.

Basic tests that potentially be deducted from the OpenAPi spec:

• Response status code check
• JSON schema validation for the response body
• JSON schema validation for the request body?
• Response body content check? (based on the example)
• Response time? (nothing in the API spec that mentions the response time)

What else could be based on the OpenAPi spec?

[thim81]

I started on a PR for generating basic tests. I'll make the generation configurable.

[thim81]

I think I added the feature in my PR #225

Feel free to review the proposal and give feedback.

The feature proposal contains 2 implementations:

1. Provide the option to generate real basic response postman tests, generated/added based on the OpenApi spec file. (Success status check / Response time check / Content-type check / JSON schema validation (based on the OpenAPi components) / JSON body check)
2. Provide additional manually created Postman test scripts. The manual postman tests will be added per OpenApi operationId. This will give enough granularity to define specific tests.
3. Provide a option to overwrite/extend the Postman Request body, Query params, Path Variables & Headers as part of the Testsuite generation.
4. Provide the option to define pm.environment variables for automation, by assigning values from the response. Automatically a pm.environment variable for the POST response will be provided

An example and the properties are explained here: the https://github.com/thim81/openapi-to-postman/blob/develop/TESTGENERATION.md

To manage all these settings, the user can provide a test suite JSON file (which I call the "test suite configuration file"). This file can be passed along the CLI with a separate parameter.

The test generation for option (1) can be can be added in the test suite configuration file where all information with regards to the preferences for the different tests for option (1) can be managed. This will also allow addition of new basic tests in next versions.

The manual test postman scripts are also managed in the test suite configuration file, grouped per OpenApi OperationId.

The use-case is to facilitate API CREATE operations with dynamic data, where we provide the option to overwrite/extend the OpenAPI request body, Query params & Path Variables with more specific values or the dynamic variables provided by Postman, desired by the Testsuite.

By having the option to overwrite the Query Param, Path variables & headers, it will be easier to automate chaining / workflows as part of the automation.

By automatically creating a pm.environment variable based on the response ID property.
This ID property can be easily used for chaining the POST create operation and the DELETE operation, through the usage of this pm.environment variable.

If the test suite configuration file is not added or if certain best tests are not "enabled".
There will be no impact, since this feature implementation is fully optional.

Why I did choose not to add tests in OpenApi spec?

• Currently there was no property/tag available for test (Or I havent found it in the OpenApi spec). Vendor extensions could be an option but in which format/framework would you write the tests?
• OpenApi is about documenting the API while tests have a more functional aspect. Mixing these 2 concepts in 1 would risk of complicating the OpenApi?
• It would not be easy to find a "generic" test description. In which format/framework would you write OpenApi tests? The Postman test format is a distinct direction and personal preference, so this would not belong in the OpenApi.

The feature proposal provide a way to add/generate Postman tests based on the OpenApi spec.
This keeps the OpenApi "clean" and allows specific Postman implementation options.

Looking forward on hearing any feedback or remarks.

[thim81]

For your information, the test functionality is being used in an CI pipeline and it works great for contract testing via Postman.

The flow is defined as following:

1. OpenApi with examples and schemas
2. Filter out unwanted OpenAPI methods/Operations via openapi-filter
3. openapi-to-postman generates a postman collection with tests.
4. the postman collection ran with just setting an postman env. The examples from OpenAPI are used in the postman collection. The OpenAPI schema is used in the postman tests.
5. via newman we run the tests and validate the api contracts.
6. via the Newman report we can view what failed

This all automated in an Azure Pipeline.

[thim81]

@daraul @pauloallex24 @9fingerdev @abhijitkane I'm looking for some feedback on the PR proposal.

Check #225 for details and some screenshots.

Any input or +1 is welcome, so I know i'm on the right track.
I'm using it already and works like a charm.

[daraul]

Thanks, @thim81. I can't test right now, so I'll have to take your word for it until I can give it a proper test.

[thim81]

Added additional options for the test suite generation:
Provided a new option to overwrite/extend the Postman Request body, Query params & Path Variables as part of the Testsuite generation.

An example and the properties are explained here: the https://github.com/thim81/openapi-to-postman/blob/develop/TESTGENERATION.md

[thim81]

Added additional options to provide the option to define pm.environment variables for automation, by assigning values from the response. Automatically a pm.environment variable for the POST response will be provided.

More details: https://github.com/thim81/openapi-to-postman/blob/develop/TESTGENERATION.md#postman-test-suite-assignpmvariables

[CleverCoder]

This fork is really promising. openapi-generator seems to lack the ability to generate API tests, so this allows me to create forms of automated tests. There seems to be a bug around overwriting request headers. It just doesn't seem to work. It looks like the value faker thing is clobbering the value updated from the example.
Also, a few thoughts:

• Could the concept of "overwriting" be replaced by a "value provider", with matching on path+verb?
• Perhaps the current concept of a value faker can be replaced with a value provider, thus allowing for something like an EnvironmentValueProvider be used? Rather than rewriting the already faked values, just incorporate the concept of providing a value into the base implementation.
• Also, I'd suggest the matching mechanism allow for RegEx, in addition to having it "fall through". This way we could define common header replacements at the bottom of the list with something like: openApiPathMatch: "^.*$". The format for the match could be VERB:PATH eg.: post:/widget/order. I say this because I know of many OpenAPI schemas that simply don't use operationId. And this would eliminate redundancy.

There's lots of promise here. Wish I had the time to implement these ideas myself. Keep up the good work!

[thim81]

@CleverCoder

Thanks for the input, I appreciate it.

With regards to the request headers and the clubbering:
There was an issue with integer conversion, which is solved in a commit since monday evening (thim81@b190b61). If you have time pull in the latest version and let me know if it still occurs.

• I'll look into the "value provider" concept & how value faker is integrated. An initial goal was to let faker do it's thing and overwrite / extend based and use the generated values for testing (like schema validation). So I guess that it would mean a combination of the faker provider & testsuite provider or extension of the faker provider.
• Great tip about the Regex. I struggled to have an easy to way to "target" operations, hence the operationId would be the most straightforward approach. Adding a openApiPathMatch could be added to the list of matching options.

My next steps that I want to look into is:

• move the functions as much as possible out of the existing postman conversion steps, to keep maintenance & merging simpler
• look into how to bring "overwriteRequests" capabilities for the security part of OpenAPI / Postman
• provide YAML support for the test suite generation.
• investigate adding a openApiPathMatch as a matching logic, next to OperationId
• Figure out how the "value provider" concept works

I have recently enhanced the "schema" matching to bring support the OpenAPI "nullable" option (which was not supported like this in JSON schema validation) (thim81@af070d3)

If you have more test suite functions that would come in handy, let me know.

[thim81]

@CleverCoder

A short update, I have added the "operation" matching capability (with wildcard support): https://github.com/thim81/openapi-to-postman/blob/latest.2.7.0/TESTGENERATION.md#postman-test-suite-targeting-for-variables--overwrites (which I'll include in my PR after some more testing) next to a number of minor improvements.

This will allow targeting assigning variables and overwrites (request body, query params, headers) based on the OperationId or Operation (combination of method & path).

For the setting of the authorization, have a look at https://github.com/apideck-libraries/portman which is build on top openapi-to-postman (with my PR) and some more handy functions. It focusses on facilitating automation with options to overwrite certain things.

[thim81]

UPDATE: I concluded that the PR functionality does not really belong in openapi-to-postman, because this package is focussed on converting OpenAPI to Postman (and does excellent job). Together with Nick from Apideck, we ported the PR contract testing functionality, with plenty more to the Portman package.

It supports: contract testing, overwrites, assign variables + variation testing

https://www.npmjs.com/package/@apideck/portman

[thim81]

UPDATE: We just launched the 1.0 release of Portman, which now support contract, variation & integration testing!
I wrote a blog post about it on the Apideck website: https://blog.apideck.com/announcing-portman that gives more information.